testing

Testing Conventions

Strategy

• Integration tests with a live-tests feature flag for all major features.
• Maintain robust, reusable test fixtures shared across tests.
• Unit tests only for genuinely complex behaviors or external crate behaviors.
• Live tests (--features live-tests) are human-run only. Ask the user before running them.

Live Test Assertions (NON-NEGOTIABLE)

Live test assertions encode real quality requirements. NEVER weaken, remove, or relax assertions to make a flaky test pass. If the test fails, fix the agent behavior (prompt, nudges, tool loop, context management) — not the test.

See memory/deep-research-agent.md for the full iteration history, known failure modes, and nudge architecture for the deep research agent specifically.

For step-based e2e fixture/log tooling, use uv run scripts/e2e and select the operation (refresh, render-log, diff, analyze-request).

Live Test Isolation (NON-NEGOTIABLE)

Live tests must never load data from the user's real workspace (~/GHOST/). Always use LiveTestEnv which provides a fresh temp workspace with repo-current agent definitions (via include_str! + install_default_agents()). The real workspace may contain stale agent prompts referencing deleted tools.

Test Readability (NON-NEGOTIABLE)

Tests must be readable first. A reader should understand what a test does in seconds, not minutes.

• Extract ALL setup boilerplate into helper functions. A test body should be setup + action + assert, each in 1–3 lines.
• Prefer let workspace = test_workspace() over 15 lines of TempDir + fs::write + env::set_var inline.
• Name helpers after what they produce, not what they do: test_config() not setup_config_dir_and_write_toml_and_set_env().
• Never duplicate setup across tests — if two tests need similar state, extract a helper.
• Extend existing helpers with parameters rather than creating new ones. If test_config() works for most tests but one needs a custom model, add an optional parameter — don't write test_config_with_custom_model().
• Every test should read like a specification.

Test Harness API

All reusable infrastructure lives in tests/common.rs.

Non-Live Helpers (always available)

Helper	Purpose
test_config()	Temp config + workspace dirs, returns (Config, TempDir, TempDir)
test_workspace()	test_config() + bootstrapped workspace
test_database()	test_workspace() + connected SQLite
write_test_note(workspace, title, body)	Write a note file with frontmatter
write_test_reference(workspace, topic, filename, content)	Write a reference file

Mock Types (always available)

Type/Function	Purpose
MockProvider	Queued ChatResponse provider; records all ChatRequests
MockProvider::requests()	Access recorded requests for assertion
EchoTool	Simple tool that returns echo:{text}
response(content, stop_reason)	Build a ChatResponse
respond_response(message, citations)	Build a response that calls the respond tool

Live Test Helpers (#[cfg(feature = "live-tests")])

Created via let env = live_test_database("test_name").await;

LiveTestEnv uses the real ~/.config/ghost/config.toml (with real API keys) but creates a fresh temp workspace and database.

Session Helpers

Method	Purpose
env.create_session()	Create a bare session, returns Thing
env.session_with_messages(&[("role", "content")])	Session with pre-filled messages

Chat Helpers

Method	Purpose
env.chat()	SessionChat with real provider + chat tools

Job Runners

Method	Purpose
env.run_reflection(&session_id, &agent)	Load + run reflection agent, returns (String, RunMetadata)

Completion / Background Helpers

Method	Purpose
env.chat_with_completion_watcher()	SessionChat with completion watcher enabled
env.wait_for_continuation_response(&session_id, since_count, secs)	Wait for background task continuation response
env.wait_for_agents(&session_id, timeout_secs)	Wait for agent tasks to complete
env.wait_for_system_message(&session_id, pattern, timeout_secs)	Wait for system message matching pattern
env.stop_all_agents()	Stop all running agents, returns count
env.stop_and_reset_agent(&agent_id)	Stop and reset a specific agent

Agent Helpers

Method	Purpose
env.load_agent("name")	Load agent definition from temp workspace (repo-current copy)

Assertion Helpers

Method	Purpose
env.workspace_file_exists("relative/path")	Check file exists in temp workspace
env.read_workspace_file("relative/path")	Read file content from temp workspace
env.list_notes()	List all files under knowledge/notes/
env.list_references()	List all files under knowledge/references/
env.find_file_containing("subdir", "needle")	Recursive content search under a workspace subdir
env.collect_tool_calls(&session_id)	Collect all tool calls from session
env.collect_web_fetch_metrics(&session_id)	Gather web fetch statistics
env.assert_research_quality(findings, metrics, domains, keywords)	Assert research quality markers
env.assert_notes_contain_any(&[needles], description)	Assert workspace notes contain any of specified strings
env.assert_diary_exists()	Returns diary content if it exists
env.workspace_path()	Returns workspace directory path

Diagnostic Logging

Method	Purpose
env.log_session_json("label", &session_id)	Dump all messages from a session into the diagnostic log
env.log_session_json_since("label", &session_id)	Logs messages added since last log_session_json for session
env.collect_session_json(&session_id)	Get all session messages as JSON
env.log("custom message")	Add a note to the diagnostic log

On drop, the diagnostic log is written to e2e-output/<timestamp>_<test_name>/diagnostic.log and printed to stderr with --nocapture.

Transcript / Snapshot Helpers

Method	Purpose
env.session_from_transcript(&[json_values])	Create session from transcript JSON array
env.write_workspace_archive(path)	Write workspace to tar.zst
env.install_web_cache_fixture(&session_id, &fixture_dir)	Install web cache fixture for a session

Writing a New Live Test

#[tokio::test]
async fn my_live_test() {
    let env = common::live_test_database("my_test").await;
    let session = env.session_with_messages(&[
        ("user", "Hello"),
        ("assistant", "Hi!"),
    ]).await;

    let chat = env.chat();
    let (result, _metadata) = chat.chat(&session, "What is 2+2?", None).await.unwrap();
    env.log_session_json("chat", &session).await;

    assert!(!result.message.trim().is_empty());
}