verify-sdk

Verify the OpenHands SDK in-process

The "user" here is a Python program that imports openhands.sdk and runs an agent in its own process. This skill proves that program's view of the SDK: each recipe builds an agent with the public API, runs it, and checks what came out (state, files, events, payloads), with a scripted model where the model cannot matter and a real one where it can. It never reruns the unit tests: CI does that on every PR, and each row links them on its Unit: line. The agent server is a separate surface with its own skill, verify-agent-server; what the server adds on top of the SDK belongs there.

Launch

There is no server. Prepare the checkout once, then start a run per scenario:

uv sync --dev
export PATH="$PWD/.agents/skills/verify-sdk/scripts:$PATH"
export SDK_VERIFY_RUN=$(control-sdk new --field run)                     # scripted
export SDK_VERIFY_RUN=$(control-sdk new --llm deepseek-flash --field run) # live

A run has its own HOME, OH_PERSISTENCE_DIR, conversation store, workspace and tmux server, and drops the caller's LLM_* and OH_* variables, so it never touches a developer's ~/.openhands. Live runs read DEEPSEEK_API_KEY when a model is called; without it they exit 3 and report blocked.

Doctor

sdk-map check is the read-only health check: it validates the map's structure and IDs, the index counts, the baseline line, and that every path the map cites still exists. Run it first and whenever a recipe surprises you. For the environment, uv run python -c "import openhands.sdk, openhands.tools, openhands.workspace" must succeed.

Drive

control-sdk does one thing per command and prints JSON; --check FIELD OP VALUE asserts on it and --field PATH prints one value for the shell:

CID=$(control-sdk conversation new --script S05/write_and_read.json --field id)
control-sdk conversation send "$CID" --text "write notes.txt" --run --check execution_status eq finished
control-sdk conversation events "$CID" --kind ActionEvent --check count eq 4
control-sdk program S05/reload_events --arg "$CID" --check roundtrip_equal eq true
control-sdk exec --model-env -- python examples/01_standalone_sdk/01_hello_world.py

Conversations persist in the run, so each command resumes the one before. Programs under programs/ are short scripts like the examples that print one JSON result. references/recipes.md explains scripts, programs, live runs and conventions.

Each family file in the map lists its sub-features and a Driving it section of bullets, each naming the IDs it proves and carrying sh blocks. Run them literally from the repository root, or let the tool replay them:

sdk-map run --file S05                                  # one family
sdk-map run --file S05 --only S05.event-base            # the bullets that name these IDs
sdk-map run --all --out "$SDK_VERIFY_HOME/full"         # the whole map

A bullet passes when every block it ran exits 0; exit 3 means blocked. A block that needs a real model (a live run, --model-env, an example script or the integration runner) runs only with DEEPSEEK_API_KEY set and is skipped whole otherwise. An integration run passes only when its results.json shows every requested test succeeded, because the runner exits 0 on failed tests. Each block runs in its own process group, which is killed when the block ends or times out.

Evidence

sdk-map run writes one ledger row per sub-feature to <out>/ledger.jsonl and each bullet's transcript to <out>/transcripts/. <out> is a new directory under $SDK_VERIFY_HOME (default $TMPDIR/sdk-verify) unless --out names one. sdk-map report --runs <out> [...] renders the latest result per sub-feature, fail first; --baseline <earlier run> lists what changed since then.

Proof standards: drive the public API a program uses, not internal setters or the unit tests; assert state, files and events, never a model's wording; a scripted model (TestLLM) through the real agent loop is the deterministic proof, a live run, example or integration run the real-model one, and a row says which it got (scripted, live, example, integration). Unreachable is blocked with the missing prerequisite, never a pass.

Cleanup

Recipes write into their run's workspace, under $SDK_VERIFY_HOME (default $TMPDIR/sdk-verify), never the checkout. Keep the run directories while the evidence matters; git status should show nothing a recipe created.

Workflows

• Verify a change or a PR: sdk-map owners --changed origin/main...HEAD maps the changed paths to the families whose Source: lines own them, and lists product paths no family claims (a map gap). Run those families with sdk-map run, with DEEPSEEK_API_KEY set for the live bullets, and report per the agent-server report contract.
• Run the daily pass: follow references/daily.md: sdk-map check, the families changed since the Maintenance baseline line (sdk-map owners with no arguments), and nothing else: families nothing in the range touched wait for the weekly pass, and a pass with no new commits on main stops early.
• Maintain the map (periodic or weekly): follow references/maintenance.md: a frozen BASE/TARGET, a PR-intent ledger, a source wave, a full replay with a model key, and at most one PR of proven corrections.

Triage what you find

• Map drift: the map describes something the SDK no longer does by design. Fix the entry with current source and a passing recipe.
• Harness gap: the SDK works but no recipe shows it yet. Write the program or control-sdk step that does; adding unit tests to the product is a separate PR.
• Product bug: the SDK is broken. Keep the transcript, search existing issues, and file it in this repository; mark the bullet's expected result with Known failure (reproduced <date>, #<issue>). Never rewrite an expected result to bless broken behavior, and never fix product code inside a pass.
• Blocked: name the missing prerequisite (key, Docker, Chromium, account, interactive login) and the command attempted.

See references/adaptation.md for where these ideas come from.