add-harness
Add a new agent harness (opencode, Gemini CLI, Codex, Amp, a local Ollama-backed runner, or any CLI that loads skills) to a shakedown config so its skill conformance can be measured, and diagnose a harness that `shakedown doctor` reports as not qualifying. Use this whenever someone wants shakedown to run against a harness it does not yet support, asks why a harness fails doctor, asks how to point an existing harness at a different model or provider, or is editing the `[harness.*]` block of a shakedown.toml. Use it even when they only name the tool ("can this test opencode?") without mentioning config or TOML.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-21
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- add-harness
- description
- Add a new agent harness (opencode, Gemini CLI, Codex, Amp, a local Ollama-backed runner, or any CLI that loads skills) to a shakedown config so its skill conformance can be measured, and diagnose a harness that `shakedown doctor` reports as not qualifying. Use this whenever someone wants shakedown to run against a harness it does not yet support, asks why a harness fails doctor, asks how to point an existing harness at a different model or provider, or is editing the `[harness.*]` block of a shakedown.toml. Use it even when they only name the tool ("can this test opencode?") without mentioning config or TOML.
03 SKILL.md 正文
Adding a harness to shakedown
A harness is described entirely by config. No code is written to support
one, so this task is: find the flags, write the block, run doctor, and fix
whatever it names.
The reason doctor exists is that a harness can look configured and still
be measuring nothing. Every prerequisite below is verified by watching what
the agent actually did, never by reading a help string or an inventory
command.
Step 0: Confirm the binary is here
Everything below is read off the CLI in front of you, so start by establishing there is one:
command -v <cli> && <cli> --version
If it is not installed, stop and say so. shakedown runs harnesses; it does
not install them, and the block you would write without the binary is
guesswork. Flags drift between versions, an agent's memory of a CLI is
frequently a version or two stale, and a plausible wrong flag produces a
config that fails at doctor in a way that looks like the harness's fault
rather than the block's. Ask the operator to install it, or work on a
harness that is present.
The version matters beyond mere presence: record it, because it is part of what any measurement made with this block will mean.
Step 1: Check the binary can do the five things
A harness qualifies if it can do these. Four are required.
| # | prerequisite | required | if missing |
|---|---|---|---|
| 1 | run headless from a prompt and exit on its own | yes | unusable |
| 2 | load a skill from a directory you control | yes | unusable |
| 3 | emit machine-readable output with tool calls and text | yes | unusable, unless it can export the transcript afterwards (Step 3) |
| 4 | continue a session | no | inputs_resolved reports unsupported |
| 5 | run without a TTY | yes | unusable |
| 6 | what else the model could see | reported only | nothing |
Read the binary's --help and find the flag for each. What you are looking
for, in the usual naming:
- A non-interactive or print flag (
-p,--prompt,run). - An output format flag naming JSON or stream-json. Newline-delimited JSON is what shakedown parses.
- A session identifier flag, and a resume flag that takes it.
- A model flag, so the same harness can be measured against several models.
- A permission or approval flag, discussed in Step 4.
Prerequisite 4 being optional is deliberate. A harness must never fail a dimension it cannot physically support, so a harness with no resume is reported as unsupported with a reason, and its other checks stay comparable against everything else.
Step 2: Write the block
[harness.my-harness]
start = [
"my-agent", "run", "{prompt}",
"--json",
"--model", "{model}",
"--session", "{sid}",
]
resume = [
"my-agent", "run", "{reply}",
"--json",
"--model", "{model}",
"--resume", "{sid}",
]
skills = ".my-agent/skills"
activation_tool = "Skill"
# Exactly one of these, and only for `--sandbox container`:
image = "ghcr.io/you/my-agent:1.2.3"
# dockerfile = "docker/my-agent.Dockerfile"
# context = "." # with `dockerfile`, to COPY from the repo root
[harness.my-harness.env]
MY_AGENT_TOKEN = "${MY_AGENT_TOKEN}"
[[matrix]]
harness = "my-harness"
models = ["some-model"]
Commands are lists, not strings. Each token is one argument, so a prompt
containing a quote or a leading dash stays a prompt and can never become a
flag. Substitution slots are {prompt}, {reply}, {model}, and {sid}.
skills is the directory this harness scans for skills, relative to the
workspace. The skill under test is copied to <skills>/<name>/. Use the
harness's own discovery path, the one it would find a skill in with no flags
at all.
activation_tool is the substring identifying a skill-activation call
in this harness's output, Skill on Claude Code and activate_skill on
Gemini. It is what separates "the skill never fired" from "it fired and
failed". Without it a run where the model ignored the skill and improvised
scores as an ordinary failure, and the number then describes the base model
rather than the skill.
env is declared-only. The sandbox gets these variables and nothing
else, which is what makes host contamination structurally impossible rather
than something a flag suppresses. Values come from the host through ${VAR}
so the TOML holds references, never secrets.
image or dockerfile is needed only for --sandbox container, and
declaring both is refused. Use image when you already build and push one,
and dockerfile when you would rather keep the environment in the repo.
Either way it holds the harness plus whatever the skills need at runtime,
because a skill whose bin/ cannot execute fails checks for reasons that
have nothing to do with the skill. See examples/docker/ for one per
shipped harness.
The path is relative to shakedown.toml, and it is built once per run
rather than per scenario.
context is the directory a dockerfile build can COPY from,
defaulting to the dockerfile's own. Widen it with context = "." when the
image needs something this repo builds rather than something it installs
from a registry — your own CLI, say. Without it the COPY is outside the
context and Docker refuses it.
Step 3: Tell shakedown where the tool calls are
Harnesses put tool calls at different depths. This is the one shape question, and it is config rather than code.
Run the harness once by hand with its JSON flag and look at a line containing a tool call. Then choose:
- Nested: the call sits inside a list under a dotted path, as Claude
Code's does in
message.content[]. Setcontainer = "message.content". - Flat: the call is a top-level record, as Gemini CLI's is. Omit
containerentirely.
Since this is a genuine fork with a short list of answers, present it as a decision rather than a prose question, and show the harness's own output as the preview so the choice is made against evidence.
The remaining keys are renames, and the defaults match Claude Code:
[harness.my-harness.events]
container = "" # dotted path to a list; omit if flat
discriminator = "type" # the key naming a block's kind
tool_marker = "tool_use" # that key's value on a tool call
name_key = "name"
args_key = "input"
text_key = "text"
text_marker = "text"
The four that name a key — discriminator, name_key, args_key,
text_key — may each be a dotted path, which covers a flat record that
still buries what you need. opencode's tool call is top-level but carries
its name and arguments under a part object, so it reads name_key = "part.tool" and args_key = "part.state.input". (tool_marker and
text_marker are values compared against, not keys, so a dot in one does
nothing.)
A harness matching neither shape fails at doctor step 3, loudly, instead
of silently scoring zero everywhere.
When the harness prints prose and nothing else
Some agents stream no events at all. Hermes prints only its final answer,
so there is nothing to parse while it works. Before writing it off, check
whether it can hand over the transcript afterwards: hermes sessions export --format trace writes Claude Code JSONL, and a start command may
be two commands.
start = [
"sh", "-c",
"hermes -z \"$1\" -m \"$2\" >&2 && hermes sessions export --format trace --no-redact -",
"hermes", "{prompt}", "{model}",
]
The prompt stays a positional argument ($1) rather than being pasted
into the script, so it is still exactly one argument and still cannot
become a flag. Check what the export does to the transcript on the way
out, too: Hermes redacts tool arguments unless told not to, and those
arguments are what tool_used reads. Send the harness's own prose to
stderr rather than /dev/null: a run that fails for its own reasons then
says why, instead of scoring as a skill that never fired.
Step 4: Let the harness run the tool
Most harnesses refuse tool calls by default, and a refusal still appears in the transcript as a call. A config that omits the permission flag therefore produces runs where the CLI was requested, denied, and never executed.
shakedown reads the harness's denial records and fails tool_used with "was
requested but denied", so this shows up as a diagnosis rather than a silent
wrong number. Fix it in start and resume by allowing the tools the skill
needs.
Watch for a permission mode that covers file writes but not shell commands.
Claude Code's acceptEdits allows Write while still prompting for Bash,
so a skill that shells out to its CLI is denied under it.
Step 5: Run doctor until it qualifies
shakedown doctor --harness my-harness
doctor runs a canary skill that asks one question, then runs
echo shakedown-ok. Seeing that shell call is only possible if the harness
ran headless, discovered the skill, surfaced it to the model, followed it,
and emitted parseable output, so one cheap task settles prerequisites 1, 2,
3 and 5 at once. The question is there to force a second turn, which
settles 4.
my-harness
# prerequisite detail
1 headless run ok exit 0
2 skill surfaced at runtime FAIL never activated
3 output parsed ok 1 tool calls, 3 texts
4 session resume ok 2 turns
5 no TTY required ok ran without a terminal
6 environment visibility ok 16 other skills visible
does not qualify: blocked on skill surfaced at runtime
Work the failing number:
- 1 fails: the command needs a headless flag, or it is waiting on input. A hang shows as a timeout rather than a non-zero exit.
- 2 fails with "never activated":
skillspoints somewhere the harness does not scan. Check the canary landed there, in the workspacedoctorprints at the end. - 2 fails with "activated but never ran": discovery works and permission does not. Go back to Step 4.
- 3 fails: the output is not newline-delimited JSON, or
eventsdoes not match its shape. Go back to Step 3. - 4 fails: the resume flag did not carry the session. Some harnesses
resume by literal
latestrather than an id, which is safe here because the sandbox holds exactly one session. If the harness genuinely cannot resume, delete theresumeline and let the check reportunsupported. - 6 never blocks. It reports how many other skills the model could see,
which on the
tmpsandbox includes everything installed on the host. Use--sandbox containerwhen that contamination matters.
If doctor prints "has no [[matrix]] entry, so there is no model to
check it with", the harness is described but nothing names a model for it.
Add a [[matrix]] entry, or pass --model for a one-off check.
Qualifying on tmp does not mean qualifying in a container
The sandbox is part of what you are configuring, and prerequisite 2 is a
different question in each. A harness can surface the skill perfectly on
tmp and never surface it in a container, because the container has a
different filesystem, a different HOME, and none of the host's
configuration. Every run then reports "never activated" and the skill
looks broken when the sandbox is at fault.
So if you intend to use --sandbox container, qualify it there too —
but only once the credential is declared, because a container inherits
nothing:
[harness.my-harness.env]
MY_AGENT_TOKEN = "${MY_AGENT_TOKEN}"
Without that, doctor --sandbox container fails on authentication, which
tells you nothing about discovery and is easy to misread as a harness that
does not qualify. Get the credential in first, confirm the harness runs at
all in the image, and only then trust what prerequisite 2 says there. A
subscription login is the awkward case: on macOS it lives in the Keychain
rather than in a file, so no amount of mounting carries it in and the
harness needs a token it can read from the environment.
Pointing an existing harness at a different provider
Same binary, different model behind it. This is what separates harness quality from model quality, so it is worth setting up.
[[matrix]]
harness = "claude-code"
label = "claude-code/glm-4.6"
models = ["glm-4.6"]
env = { ANTHROPIC_BASE_URL = "https://open.bigmodel.cn/api/anthropic",
ANTHROPIC_AUTH_TOKEN = "${GLM_TOKEN}" }
Give it a label. An env override changes what is behind the harness, and
comparing two backends under one name would silently average them together.
Two traps worth naming
Do not reach for an isolation flag that disables skills. Claude Code's
--safe-mode lists skills among the customizations it turns off, so a run
using it never shows the skill to the model, and the result reads as "this
harness ignores instructions". Isolation belongs to the sandbox. Use
--sandbox container.
A static inventory does not prove runtime visibility.
claude --plugin-dir X plugin details reported Skills (1) for a run whose
init event showed the skill absent from the model's list. That is why the
skill is copied into the harness's own discovery directory, and why doctor
asserts on what the model did rather than on what a subcommand claims.
04 同仓库更多 Skills
fake-skill
A skill used only by shakedown's own offline tests.
shakedown-canary
Run the shakedown canary check. Use whenever someone asks to run the canary check, verify the harness, or confirm shakedown is wired up correctly.
create-cases
Author a shakedown cases file (`shakedowns/<slug>.cases.toml`) for an existing agent skill: read the skill, turn what it promises into cases, supply fixtures for the commands it must not really run, and check the result with `shakedown case validate`. Use this whenever someone wants to measure, smoke-test, or add conformance cases for a skill, asks "how do I test this skill", points at a SKILL.md and wants cases written, or is editing a `.cases.toml`. Use it even when they only name the skill ("write cases for connect-agent-to-hub") without mentioning shakedown or TOML.
analyze-results
Read a shakedown report (`shakedown-report.json`) and say which file to open: the measured skill's own `SKILL.md`, the `shakedowns/<slug>.cases.toml` holding its cases, the harness block in `shakedown.toml`, or a flag on the run itself. Use this whenever someone has run `shakedown case run` and wants to know what a number means or what to do about it, pastes a report or the PR comment rendered from one, asks why `skill_fired`, `tool_used`, `artifact_created` or `inputs_resolved` came out low, asks whether a rate is worth acting on, or says a conformance run failed and they cannot tell whose fault it is. Use it even when they only quote the percentage ("my skill scores 60% on inputs_resolved, now what") without naming shakedown or the report file.
add-harness
Add a new agent harness (opencode, Gemini CLI, Codex, Amp, a local Ollama-backed runner, or any CLI that loads skills) to a shakedown config so its skill conformance can be measured, and diagnose a harness that `shakedown doctor` reports as not qualifying. Use this whenever someone wants shakedown to run against a harness it does not yet support, asks why a harness fails doctor, asks how to point an existing harness at a different model or provider, or is editing the `[harness.*]` block of a shakedown.toml. Use it even when they only name the tool ("can this test opencode?") without mentioning config or TOML.
create-cases
Author a shakedown cases file (`shakedowns/<slug>.cases.toml`) for an existing agent skill: read the skill, turn what it promises into cases, supply fixtures for the commands it must not really run, and check the result with `shakedown case validate`. Use this whenever someone wants to measure, smoke-test, or add conformance cases for a skill, asks "how do I test this skill", points at a SKILL.md and wants cases written, or is editing a `.cases.toml`. Use it even when they only name the skill ("write cases for connect-agent-to-hub") without mentioning shakedown or TOML.