[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7AAEXCDG258QRVSSFQV0MV9":3,"skill-same:01M7AAEXCDG258QRVSSFQV0MV9":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7AAEXCDG258QRVSSFQV0MV9","analyze-results",null,"Read a shakedown report (`shakedown-report.json`) and say which file to open: the measured skill's own `SKILL.md`, the `shakedowns\u002F\u003Cslug>.cases.toml` holding its cases, the harness block in `shakedown.toml`, or a flag on the run itself. Use this whenever someone has run `shakedown case run` and wants to know what a number means or what to do about it, pastes a report or the PR comment rendered from one, asks why `skill_fired`, `tool_used`, `artifact_created` or `inputs_resolved` came out low, asks whether a rate is worth acting on, or says a conformance run failed and they cannot tell whose fault it is. Use it even when they only quote the percentage (\"my skill scores 60% on inputs_resolved, now what\") without naming shakedown or the report file.","JasperHG90\u002Fshakedown","JasperHG90","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F8079803?v=4","01M7AAAKN62DYZ9XRCXAN0HGEF","Jasper Ginn","01M7AAAKP8D78B8ANVKEV7MRK8",1,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fskills\u002Fanalyze-results\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: analyze-results\ndescription: >-\n  Read a shakedown report (`shakedown-report.json`) and say which file to\n  open: the measured skill's own `SKILL.md`, the\n  `shakedowns\u002F\u003Cslug>.cases.toml` holding its cases, the harness block in\n  `shakedown.toml`, or a flag on the run itself. Use this whenever someone\n  has run `shakedown case run` and wants to know what a number means or\n  what to do about it, pastes a report or the PR comment rendered from one,\n  asks why `skill_fired`, `tool_used`, `artifact_created` or\n  `inputs_resolved` came out low, asks whether a rate is worth acting on,\n  or says a conformance run failed and they cannot tell whose fault it is.\n  Use it even when they only quote the percentage (\"my skill scores 60% on\n  inputs_resolved, now what\") without naming shakedown or the report file.\n---\n\n# Reading a shakedown report\n\nA report is a handful of percentages, and the operator's question is never\nwhat the number is. It is which file to open.\n\n`inputs_resolved: 60%` has two completely different causes. The skill\nguessed a value it was supposed to ask for, which is a defect in the\nskill's `SKILL.md`. Or the agent did ask, in words the case's `match`\npattern never covered, which is a defect in\n`shakedowns\u002F\u003Cslug>.cases.toml`. Those are different files, usually owned by\ndifferent people, and one of them is not broken at all. The report holds\nenough to tell them apart, so a diagnosis that ends at \"consider improving\nthe skill\" has thrown away the only thing that made the run worth paying\nfor. **Every finding you report ends at a path.**\n\n## The report is self-sufficient\n\nYou do not need the workspace, and you do not need the run to have been\nmade with `--keep`. `runs[].detail[].said` is an array of the text the\nagent produced on that turn, and it is there for every turn of every run,\nkept or not. That plus the `reason` on each result is the whole basis for\nthe routing below.\n\nTwo things do depend on the workspace surviving:\n\n- `runs[].workspace_kept` says whether it did. A run with a check in\n  `fail` always keeps its workspace, so anything you route below is on\n  disk. Everything else is deleted unless the operator passed `--keep`,\n  and that includes a run where the skill never activated, since\n  `not_triggered` is not a failure. The one row where you would most like\n  a workspace is the one least likely to have one.\n- `runs[].detail[].stream` points at the raw newline-delimited JSON for\n  that turn. It is written as an empty string when the workspace was\n  cleaned rather than left as a path to a directory that no longer exists,\n  so an empty `stream` means \"cleaned up\", not \"nothing was recorded\". Read\n  `said` instead, which is still there.\n\nOpen the raw stream only when `said` and the reasons leave you guessing.\nIt is the same conversation with the framing left on.\n\n## Step 1: Check whether the skill fired, before reading anything else\n\nA run where the skill never activated measured the bare model. Every other\nnumber on that row describes something you did not write, so it is not\nmerely unreliable, it is about a different subject. Settle this first and\nreport nothing else about that target until it is fixed.\n\n**`skill_fired` never reads 0%.** The check returns `pass` or\n`not_triggered`, and `not_triggered` stays out of the rate, so the column\nreads `100%` or `n\u002Fa` no matter how many runs failed to activate. Read the\ncounts, not the rate:\n\n```bash\njq '.scores | map_values(.skill_fired)' shakedown-report.json\njq '.summary.not_triggered' shakedown-report.json\n```\n\n- `not_triggered` at 0 across every target: the skill fired everywhere.\n  Move on to Step 2.\n- `rate: null` with `not_triggered` equal to that target's run count:\n  nothing fired there at all.\n- `rate: 1.0` with `not_triggered` above 0: some runs fired and some did\n  not. This is the case worth catching, because the row reads `100%` and\n  looks clean. Every other rate on it was computed over the runs that\n  fired and silently excludes the rest.\n\n`summary.ok` counts a run where the skill never activated as ok, because\n`ok` means no check reported `fail` and `not_triggered` is not a failure.\nSo a report reading `ok: 7` of 11 can be seven runs that proved something\nand can be five, and the operator's sense of how bad the run was comes\nfrom that line. Subtract `summary.not_triggered` from it before you quote\nit back.\n\nThen route it, because \"fix your description\" is wrong most of the time\nhere:\n\n| What the report shows | Open |\n|---|---|\n| No target fired, on any case | `shakedown.toml`: every harness's `skills` path and `activation_tool` |\n| One target fired, another never did | The silent harness's block alone: same two keys, read against the one that worked |\n| A target fired for one case and not another | The skill's `SKILL.md` front-matter `description` |\n\nConfirm either of the first two with `shakedown doctor --harness \u003Cname>`\nbefore editing, and read row 2 of what it prints. It runs a canary skill\nand reports whether the harness surfaced it, which separates a broken\n`skills` path from a description nobody matched.\n\nOnly the third row is a skill problem. A `description` is matched against\nthe request, so a description that is too narrow fails on the prompts it\ndoes not cover and succeeds on the ones it does. A skill the harness never\nsurfaced fails uniformly, and no amount of rewriting the description will\nmove it.\n\n## Step 2: Route each failure to a file\n\nWork from `summary.failures[]`. Each entry carries `case`, `run`, `target`,\n`failed` (the check names) and `reasons` (parallel to it). The reason\nstrings below are the ones shakedown emits, so match on them.\n\n```bash\njq '.summary.failures[] | {case, target, failed, reasons}' shakedown-report.json\n```\n\nA run usually fails two checks at once, and one of them is a consequence\nof the other. `artifact_created` in particular rarely fails on its own:\nthe file is missing because the tool was denied, or because the run\nstalled waiting on a reply. Route the run by the check that explains it\nand report the other as downstream rather than as a second finding, so the\noperator gets one file to open instead of two. When `artifact_created`\ndoes fail alone, its reason names the file and what it lacked, and that is\na question about the case's `contains` as often as about the skill: read\nthe file in the kept workspace before deciding which.\n\n### `tool_used`: \"was requested but denied\"\n\nThe harness refused the call. The skill did its job and never got to run.\n\nOpen **`shakedown.toml`**, the `start` argv for that harness, and the\n`resume` argv beside it, since a denial on turn two fails the same way.\n`runs[].detail[].denied` names what was refused, which tells you what to\nallow. For Claude Code that is `--allowedTools` and `--permission-mode`.\n\nWatch for a permission mode that covers file writes but not shell\ncommands. `acceptEdits` allows `Write` and still prompts for `Bash`, so a\nskill that shells out to its CLI is denied under it while a skill that\nwrites its own file passes. That asymmetry is why this failure gets\nmisread as a skill defect.\n\nThe other `tool_used` failure, \"no tool call mentions X\", is not this one.\nIt means no call was made at all: either the model did the work itself,\nwhich is a real finding about the skill, or the CLI was not on PATH, which\nis the operator's to fix.\n\n### `inputs_resolved`: \"no `match` fired\"\n\nA reply was still owed and nothing the agent said matched any remaining\npattern. Match the reason from its start, not from its ending: this\nsection covers only a reason opening ``no `match` fired for``. A reason\nopening `X went unsupplied` ends in the same words about nothing being\nwritten and means close to the opposite, so it is under the six-turn cap\nbelow.\n\nThe reason quotes the ending, and `runs[].unmatched_tail` holds\nthe same text unwrapped, up to the last 500 characters the agent said.\n**This is the single most useful field in the file.** Read it before\nforming any opinion.\n\n```bash\njq -r '.runs[] | select(.unmatched_tail != \"\") | \"\\(.case) run\\(.run): \\(.unmatched_tail)\"' \\\n  shakedown-report.json\n```\n\nWhat the tail says decides the file:\n\n- **The tail is a question.** The agent asked and the pattern missed it.\n  Open **`shakedowns\u002F\u003Cslug>.cases.toml`** and widen the `match` for that\n  answer until it covers the phrasing you just read.\n- **The tail is not a question.** The agent stopped, refused, or wandered.\n  Open the skill's **`SKILL.md`** and look at what it says about asking.\n- **The tail is empty.** The turn produced tool calls and no text, so\n  there was nothing to quote. A run like this does not come back from the\n  filter above, so find it by its reason and not by its tail. Go to the\n  raw `stream`, or rerun that one case.\n\nBefore widening a pattern, check it can fire at all. `match` is a TOML\nbasic string, so `match = \"(?i)\\bowner\\b\"` compiles `\\b` as a backspace\ncharacter rather than a word boundary and can never match anything. The\ncase validates, costs a real run, and fails here quoting a question the\nagent asked perfectly plainly. Doubling the backslashes is the whole fix.\n\n**shakedown deliberately does not call this a question that was never\nasked, and neither should you.** Matching cannot separate \"asked in words\nyour pattern missed\" from \"never asked, guessed instead\". The check\nreports the fact, that no pattern fired, and leaves the judgment to\nwhoever reads the tail. Preserve that when you write it up: say what the\nagent ended on and what you conclude from it, rather than reporting a\nconfident story the report does not support.\n\nNote also that for some skills, refusing to proceed and writing nothing is\nthe correct outcome. \"no `match` fired for X, and nothing was written\" can\nbe a skill behaving exactly as designed against a case that assumed it\nwould carry on. Check what the skill promises before calling this a bug.\n\n### `inputs_resolved`: the reply never reached the artifact\n\nTwo reasons say this, and they route to different files. A third reads\nalmost like the first and is neither: `PLAN.md was written with X still\nunsupplied` is the six-turn cap, below.\n\n**\"X was never supplied, yet PLAN.md was written anyway\"**. The file\narrived without the answer. The agent either guessed a value or asked past\nthe pattern and carried on regardless. Read `unmatched_tail` to tell\nwhich: a tail that announces a chosen default (\"I set the owner to\nengineering-team\") is a guess, and that is the expensive failure, because\nthe output looks right. Open the skill's **`SKILL.md`** and find what it\nsays about asking. Half the time there is no such rule and the fix is to\nwrite one. The other half the rule is already there, stated plainly, and\nwas overridden anyway: then the edit is placement, not force. Move the\nprohibition into the step where the value is actually used, because a rule\nin an earlier section has been read and passed by the time the agent is\nbuilding the command. Adding another emphatic sentence to a rule already\nbeing ignored changes nothing.\n\nIf the tail is a question instead, the `match` in the cases file missed it\n*and* the skill went ahead without an answer. Both files are at fault, and\nthe skill's half is the more serious of the two.\n\n**\"replies absent from the artifacts: X\"**. Different failure entirely.\nThe reply *was* supplied, so `runs[].replies` lists it and the harness\nasked and was answered. The answer then did not land in the file. Either\nthe skill discarded it, which is a **`SKILL.md`** problem, or the\n`contains` in **`shakedowns\u002F\u003Cslug>.cases.toml`** asserts a form the skill\nnever writes, such as a label the skill spells differently. Read the\nartifact in the kept workspace before deciding.\n\n### `inputs_resolved`: the six-turn cap\n\nTwo reasons come from here, and both mimic a section above:\n\n- `X went unsupplied, and nothing was written`\n- `PLAN.md was written with X still unsupplied`\n\nA conversation stops after six turns, which is one opening turn and five\nreplies, so a case withholding six or more facts runs out of turns with\none still owed. Nothing failed to match. The run simply ended, and the\nagent's last words were never put to a pattern at all.\n\nThat makes these the one branch where the tail carries nothing and is\n*supposed* to. `unmatched_tail` is empty by construction here, so an empty\ntail is not evidence the agent said nothing, and reading it as a silent\nstall inverts the diagnosis. Neither the `match` patterns nor the skill's\nrule about asking is implicated: no pattern was tried and no question went\nunanswered.\n\nOpen **`shakedowns\u002F\u003Cslug>.cases.toml`** and count the `[[case.answers]]`\nblocks on that case. More than five cannot all be supplied. Split the case\nor withhold fewer facts. Nothing in the skill needs to change, and a run\nthat also shows a written artifact does not mean the agent guessed: the\nfile was produced before the turns ran out.\n\n### `inputs_resolved`: \"the run timed out\"\n\nEvidence about nothing. The reason says so itself, so do not read the row\nas a skill that failed to ask.\n\nFind what hung. `runs[].detail[]` with `exit_code` of `-1` is the turn\nthat timed out, its `duration_s` sits at the ceiling, and `tool_calls` on\nthat turn shows what it was doing when the clock ran out.\n\n```bash\njq '.runs[] | select(any(.detail[]; .exit_code == -1)) | {case, run, duration_s}' \\\n  shakedown-report.json\n```\n\nTwo fixes. What the turn was doing when it stopped decides which:\n\n- **The last `tool_calls` entry shells out to a command the cases file\n  supplies.** It hung on a fixture. A double that waits on stdin, or\n  shells out to something that does, never returns and burns the whole\n  timeout on every run. Open that fixture, under the directory the cases\n  file's `fixtures` key names, and read `stderr_tail` on the turn too.\n- **Anything else.** The turn was working and 300 seconds was not enough.\n  Raise the bound, as below.\n\n`--timeout` has no flag of its own on `shakedown case run`. It bounds one\nturn rather than the whole run, so it reaches pytest after `--`:\n\n```bash\nshakedown case run .\u002Fmy-skill -- --timeout 600\n```\n\nExpect the turn to hold nothing much: no useful tool calls, `said` empty\nbecause the agent never finished a sentence, `stderr_tail` empty because\nthe harness was killed rather than failing. Say the report cannot settle\nit rather than inventing a cause, then rerun that one case at a *lower*\ntimeout. A hang caught in 60 seconds costs a fifth of one caught at the\ndefault, and a turn that was genuinely working still shows what it got\nthrough before the shorter clock stopped it.\n\n### A mixed rate resting on too few runs\n\nNot a defect, and not a file to open. This one exists so that nobody\nrewrites a working skill to chase noise.\n\n**It restrains what you say about the rate, not what you do about a\nfailure you have diagnosed.** A single run carrying `denied: [\"Bash\"]` is\na fact about that run, and it stays worth fixing whether the column above\nit reads 89% or 8%. Everything in Step 2 came from a named run and a\nquoted reason, so none of it is weakened here. What this section forbids\nis the other kind of finding: the one whose whole evidence is that a\npercentage looked low.\n\n`scores.\u003Ctarget>.\u003Cdimension>.per_case` counts the scored runs behind the\n*thinnest* case in that pool. That is the number that says whether a rate\nis worth acting on, and the `scored` count is not: five cases run twice\nalso totals ten, while being two attempts at each of five different\nthings.\n\nBeing a floor, it describes the pool rather than any one case. One case\nrun once drags `per_case` to 1 for every dimension while a sibling case\nwas tried twenty times, so read it as \"this pool contains something\nbarely tried\" and count the runs per case yourself before deciding which\ncase that was.\n\nMind the denominator too. `scored` excludes `unsupported`, so a target\nshowing `scored: 5, unsupported: 4` produced nine runs and rated five of\nthem, and `inputs_resolved: 40%` there is two passes out of five rather\nthan out of nine. Quote the rate the report computed rather than dividing\nagain from the run count.\n\n```bash\njq '.scores | map_values(with_entries(select(.value.rate != null and\n  .value.rate > 0 and .value.rate \u003C 1 and .value.per_case \u003C 10)))' shakedown-report.json\n```\n\nAnything that survives that filter is a mixed rate over fewer than ten\nruns per case. Report it as a hint, name the number, and say what it would\ntake to settle it:\n\n```bash\nshakedown case run .\u002Fmy-skill --case missing-owner --harness claude-code --repeat 20 -j 5\n```\n\nNarrow to the one case and the one target actually in question. Every\nrepeat is another live model call per case per target, so widening the\nmatrix twenty times buys mostly numbers nobody asked about. Ten is where\nthe caution stops because that is where a live matrix stops being cheap,\nnot because ten runs make a rate trustworthy: eight in ten is still\nconsistent with anything from about half the time to almost always.\n\nSay plainly that the rate alone warrants no edit. An operator who came\nfor a verdict will otherwise fix something, and a skill rewritten against\nfour runs is a skill rewritten against a coin flip. If Step 2 already\nrouted a diagnosed failure inside that same pool, name it here as the\nthing to fix and the rate as the thing to leave alone, so the two do not\nread as one recommendation.\n\n## Step 3: Write it up\n\nLead with the ordering, then one finding per symptom. For each one give\nthe evidence you read, the file, and the edit:\n\n```\nskill_fired: gemini-cli\u002Fgemini-2.5-pro never activated on either run\n(not_triggered: 2, rate: n\u002Fa), so nothing else on that row was measured.\nclaude-code fired on all 9.\n\n1. inputs_resolved 40% on claude-code, missing-owner run 0\n   Evidence  no `match` fired for platform-team; the agent ended on\n             \"who should be listed as the responsible party?\"\n   File      shakedowns\u002Fwrite-plan.cases.toml\n   Edit      the answer's `match` covers \"owner\", not \"responsible\n             party\". Widen the alternation.\n```\n\nTwo habits keep this honest. Quote the report rather than paraphrasing it,\nbecause the distinction between \"asked past your pattern\" and \"guessed\" is\ncarried in the agent's exact words. And when the evidence does not settle\nwhich of two files is at fault, say both and say what would separate them,\nrather than picking the likelier one and presenting it as a finding.\n","# Reading a shakedown report\n\nA report is a handful of percentages, and the operator's question is never\nwhat the number is. It is which file to open.\n\n`inputs_resolved: 60%` has two completely different causes. The skill\nguessed a value it was supposed to ask for, which is a defect in the\nskill's `SKILL.md`. Or the agent did ask, in words the case's `match`\npattern never covered, which is a defect in\n`shakedowns\u002F\u003Cslug>.cases.toml`. Those are different files, usually owned by\ndifferent people, and one of them is not broken at all. The report holds\nenough to tell them apart, so a diagnosis that ends at \"consider improving\nthe skill\" has thrown away the only thing that made the run worth paying\nfor. **Every finding you report ends at a path.**\n\n## The report is self-sufficient\n\nYou do not need the workspace, and you do not need the run to have been\nmade with `--keep`. `runs[].detail[].said` is an array of the text the\nagent produced on that turn, and it is there for every turn of every run,\nkept or not. That plus the `reason` on each result is the whole basis for\nthe routing below.\n\nTwo things do depend on the workspace surviving:\n\n- `runs[].workspace_kept` says whether it did. A run with a check in\n  `fail` always keeps its workspace, so anything you route below is on\n  disk. Everything else is deleted unless the operator passed `--keep`,\n  and that includes a run where the skill never activated, since\n  `not_triggered` is not a failure. The one row where you would most like\n  a workspace is the one least likely to have one.\n- `runs[].detail[].stream` points at the raw newline-delimited JSON for\n  that turn. It is written as an empty string when the workspace was\n  cleaned rather than left as a path to a directory that no longer exists,\n  so an empty `stream` means \"cleaned up\", not \"nothing was recorded\". Read\n  `said` instead, which is still there.\n\nOpen the raw stream only when `said` and the reasons leave you guessing.\nIt is the same conversation with the framing left on.\n\n## Step 1: Check whether the skill fired, before reading anything else\n\nA run where the skill never activated measured the bare model. Every other\nnumber on that row describes something you did not write, so it is not\nmerely unreliable, it is about a different subject. Settle this first and\nreport nothing else about that target until it is fixed.\n\n**`skill_fired` never reads 0%.** The check returns `pass` or\n`not_triggered`, and `not_triggered` stays out of the rate, so the column\nreads `100%` or `n\u002Fa` no matter how many runs failed to activate. Read the\ncounts, not the rate:\n\n```bash\njq '.scores | map_values(.skill_fired)' shakedown-report.json\njq '.summary.not_triggered' shakedown-report.json\n```\n\n- `not_triggered` at 0 across every target: the skill fired everywhere.\n  Move on to Step 2.\n- `rate: null` with `not_triggered` equal to that target's run count:\n  nothing fired there at all.\n- `rate: 1.0` with `not_triggered` above 0: some runs fired and some did\n  not. This is the case worth catching, because the row reads `100%` and\n  looks clean. Every other rate on it was computed over the runs that\n  fired and silently excludes the rest.\n\n`summary.ok` counts a run where the skill never activated as ok, because\n`ok` means no check reported `fail` and `not_triggered` is not a failure.\nSo a report reading `ok: 7` of 11 can be seven runs that proved something\nand can be five, and the operator's sense of how bad the run was comes\nfrom that line. Subtract `summary.not_triggered` from it before you quote\nit back.\n\nThen route it, because \"fix your description\" is wrong most of the time\nhere:\n\n| What the report shows | Open |\n|---|---|\n| No target fired, on any case | `shakedown.toml`: every harness's `skills` path and `activation_tool` |\n| One target fired, another never did | The silent harness's block alone: same two keys, read against the one that worked |\n| A target fired for one case and not another | The skill's `SKILL.md` front-matter `description` |\n\nConfirm either of the first two with `shakedown doctor --harness \u003Cname>`\nbefore editing, and read row 2 of what it prints. It runs a canary skill\nand reports whether the harness surfaced it, which separates a broken\n`skills` path from a description nobody matched.\n\nOnly the third row is a skill problem. A `description` is matched against\nthe request, so a description that is too narrow fails on the prompts it\ndoes not cover and succeeds on the ones it does. A skill the harness never\nsurfaced fails uniformly, and no amount of rewriting the description will\nmove it.\n\n## Step 2: Route each failure to a file\n\nWork from `summary.failures[]`. Each entry carries `case`, `run`, `target`,\n`failed` (the check names) and `reasons` (parallel to it). The reason\nstrings below are the ones shakedown emits, so match on them.\n\n```bash\njq '.summary.failures[] | {case, target, failed, reasons}' shakedown-report.json\n```\n\nA run usually fails two checks at once, and one of them is a consequence\nof the other. `artifact_created` in particular rarely fails on its own:\nthe file is missing because the tool was denied, or because the run\nstalled waiting on a reply. Route the run by the check that explains it\nand report the other as downstream rather than as a second finding, so the\noperator gets one file to open instead of two. When `artifact_created`\ndoes fail alone, its reason names the file and what it lacked, and that is\na question about the case's `contains` as often as about the skill: read\nthe file in the kept workspace before deciding which.\n\n### `tool_used`: \"was requested but denied\"\n\nThe harness refused the call. The skill did its job and never got to run.\n\nOpen **`shakedown.toml`**, the `start` argv for that harness, and the\n`resume` argv beside it, since a denial on turn two fails the same way.\n`runs[].detail[].denied` names what was refused, which tells you what to\nallow. For Claude Code that is `--allowedTools` and `--permission-mode`.\n\nWatch for a permission mode that covers file writes but not shell\ncommands. `acceptEdits` allows `Write` and still prompts for `Bash`, so a\nskill that shells out to its CLI is denied under it while a skill that\nwrites its own file passes. That asymmetry is why this failure gets\nmisread as a skill defect.\n\nThe other `tool_used` failure, \"no tool call mentions X\", is not this one.\nIt means no call was made at all: either the model did the work itself,\nwhich is a real finding about the skill, or the CLI was not on PATH, which\nis the operator's to fix.\n\n### `inputs_resolved`: \"no `match` fired\"\n\nA reply was still owed and nothing the agent said matched any remaining\npattern. Match the reason from its start, not from its ending: this\nsection covers only a reason opening ``no `match` fired for``. A reason\nopening `X went unsupplied` ends in the same words about nothing being\nwritten and means close to the opposite, so it is under the six-turn cap\nbelow.\n\nThe reason quotes the ending, and `runs[].unmatched_tail` holds\nthe same text unwrapped, up to the last 500 characters the agent said.\n**This is the single most useful field in the file.** Read it before\nforming any opinion.\n\n```bash\njq -r '.runs[] | select(.unmatched_tail != \"\") | \"\\(.case) run\\(.run): \\(.unmatched_tail)\"' \\\n  shakedown-report.json\n```\n\nWhat the tail says decides the file:\n\n- **The tail is a question.** The agent asked and the pattern missed it.\n  Open **`shakedowns\u002F\u003Cslug>.cases.toml`** and widen the `match` for that\n  answer until it covers the phrasing you just read.\n- **The tail is not a question.** The agent stopped, refused, or wandered.\n  Open the skill's **`SKILL.md`** and look at what it says about asking.\n- **The tail is empty.** The turn produced tool calls and no text, so\n  there was nothing to quote. A run like this does not come back from the\n  filter above, so find it by its reason and not by its tail. Go to the\n  raw `stream`, or rerun that one case.\n\nBefore widening a pattern, check it can fire at all. `match` is a TOML\nbasic string, so `match = \"(?i)\\bowner\\b\"` compiles `\\b` as a backspace\ncharacter rather than a word boundary and can never match anything. The\ncase validates, costs a real run, and fails here quoting a question the\nagent asked perfectly plainly. Doubling the backslashes is the whole fix.\n\n**shakedown deliberately does not call this a question that was never\nasked, and neither should you.** Matching cannot separate \"asked in words\nyour pattern missed\" from \"never asked, guessed instead\". The check\nreports the fact, that no pattern fired, and leaves the judgment to\nwhoever reads the tail. Preserve that when you write it up: say what the\nagent ended on and what you conclude from it, rather than reporting a\nconfident story the report does not support.\n\nNote also that for some skills, refusing to proceed and writing nothing is\nthe correct outcome. \"no `match` fired for X, and nothing was written\" can\nbe a skill behaving exactly as designed against a case that assumed it\nwould carry on. Check what the skill promises before calling this a bug.\n\n### `inputs_resolved`: the reply never reached the artifact\n\nTwo reasons say this, and they route to different files. A third reads\nalmost like the first and is neither: `PLAN.md was written with X still\nunsupplied` is the six-turn cap, below.\n\n**\"X was never supplied, yet PLAN.md was written anyway\"**. The file\narrived without the answer. The agent either guessed a value or asked past\nthe pattern and carried on regardless. Read `unmatched_tail` to tell\nwhich: a tail that announces a chosen default (\"I set the owner to\nengineering-team\") is a guess, and that is the expensive failure, because\nthe output looks right. Open the skill's **`SKILL.md`** and find what it\nsays about asking. Half the time there is no such rule and the fix is to\nwrite one. The other half the rule is already there, stated plainly, and\nwas overridden anyway: then the edit is placement, not force. Move the\nprohibition into the step where the value is actually used, because a rule\nin an earlier section has been read and passed by the time the agent is\nbuilding the command. Adding another emphatic sentence to a rule already\nbeing ignored changes nothing.\n\nIf the tail is a question instead, the `match` in the cases file missed it\n*and* the skill went ahead without an answer. Both files are at fault, and\nthe skill's half is the more serious of the two.\n\n**\"replies absent from the artifacts: X\"**. Different failure entirely.\nThe reply *was* supplied, so `runs[].replies` lists it and the harness\nasked and was answered. The answer then did not land in the file. Either\nthe skill discarded it, which is a **`SKILL.md`** problem, or the\n`contains` in **`shakedowns\u002F\u003Cslug>.cases.toml`** asserts a form the skill\nnever writes, such as a label the skill spells differently. Read the\nartifact in the kept workspace before deciding.\n\n### `inputs_resolved`: the six-turn cap\n\nTwo reasons come from here, and both mimic a section above:\n\n- `X went unsupplied, and nothing was written`\n- `PLAN.md was written with X still unsupplied`\n\nA conversation stops after six turns, which is one opening turn and five\nreplies, so a case withholding six or more facts runs out of turns with\none still owed. Nothing failed to match. The run simply ended, and the\nagent's last words were never put to a pattern at all.\n\nThat makes these the one branch where the tail carries nothing and is\n*supposed* to. `unmatched_tail` is empty by construction here, so an empty\ntail is not evidence the agent said nothing, and reading it as a silent\nstall inverts the diagnosis. Neither the `match` patterns nor the skill's\nrule about asking is implicated: no pattern was tried and no question went\nunanswered.\n\nOpen **`shakedowns\u002F\u003Cslug>.cases.toml`** and count the `[[case.answers]]`\nblocks on that case. More than five cannot all be supplied. Split the case\nor withhold fewer facts. Nothing in the skill needs to change, and a run\nthat also shows a written artifact does not mean the agent guessed: the\nfile was produced before the turns ran out.\n\n### `inputs_resolved`: \"the run timed out\"\n\nEvidence about nothing. The reason says so itself, so do not read the row\nas a skill that failed to ask.\n\nFind what hung. `runs[].detail[]` with `exit_code` of `-1` is the turn\nthat timed out, its `duration_s` sits at the ceiling, and `tool_calls` on\nthat turn shows what it was doing when the clock ran out.\n\n```bash\njq '.runs[] | select(any(.detail[]; .exit_code == -1)) | {case, run, duration_s}' \\\n  shakedown-report.json\n```\n\nTwo fixes. What the turn was doing when it stopped decides which:\n\n- **The last `tool_calls` entry shells out to a command the cases file\n  supplies.** It hung on a fixture. A double that waits on stdin, or\n  shells out to something that does, never returns and burns the whole\n  timeout on every run. Open that fixture, under the directory the cases\n  file's `fixtures` key names, and read `stderr_tail` on the turn too.\n- **Anything else.** The turn was working and 300 seconds was not enough.\n  Raise the bound, as below.\n\n`--timeout` has no flag of its own on `shakedown case run`. It bounds one\nturn rather than the whole run, so it reaches pytest after `--`:\n\n```bash\nshakedown case run .\u002Fmy-skill -- --timeout 600\n```\n\nExpect the turn to hold nothing much: no useful tool calls, `said` empty\nbecause the agent never finished a sentence, `stderr_tail` empty because\nthe harness was killed rather than failing. Say the report cannot settle\nit rather than inventing a cause, then rerun that one case at a *lower*\ntimeout. A hang caught in 60 seconds costs a fifth of one caught at the\ndefault, and a turn that was genuinely working still shows what it got\nthrough before the shorter clock stopped it.\n\n### A mixed rate resting on too few runs\n\nNot a defect, and not a file to open. This one exists so that nobody\nrewrites a working skill to chase noise.\n\n**It restrains what you say about the rate, not what you do about a\nfailure you have diagnosed.** A single run carrying `denied: [\"Bash\"]` is\na fact about that run, and it stays worth fixing whether the column above\nit reads 89% or 8%. Everything in Step 2 came from a named run and a\nquoted reason, so none of it is weakened here. What this section forbids\nis the other kind of finding: the one whose whole evidence is that a\npercentage looked low.\n\n`scores.\u003Ctarget>.\u003Cdimension>.per_case` counts the scored runs behind the\n*thinnest* case in that pool. That is the number that says whether a rate\nis worth acting on, and the `scored` count is not: five cases run twice\nalso totals ten, while being two attempts at each of five different\nthings.\n\nBeing a floor, it describes the pool rather than any one case. One case\nrun once drags `per_case` to 1 for every dimension while a sibling case\nwas tried twenty times, so read it as \"this pool contains something\nbarely tried\" and count the runs per case yourself before deciding which\ncase that was.\n\nMind the denominator too. `scored` excludes `unsupported`, so a target\nshowing `scored: 5, unsupported: 4` produced nine runs and rated five of\nthem, and `inputs_resolved: 40%` there is two passes out of five rather\nthan out of nine. Quote the rate the report computed rather than dividing\nagain from the run count.\n\n```bash\njq '.scores | map_values(with_entries(select(.value.rate != null and\n  .value.rate > 0 and .value.rate \u003C 1 and .value.per_case \u003C 10)))' shakedown-report.json\n```\n\nAnything that survives that filter is a mixed rate over fewer than ten\nruns per case. Report it as a hint, name the number, and say what it would\ntake to settle it:\n\n```bash\nshakedown case run .\u002Fmy-skill --case missing-owner --harness claude-code --repeat 20 -j 5\n```\n\nNarrow to the one case and the one target actually in question. Every\nrepeat is another live model call per case per target, so widening the\nmatrix twenty times buys mostly numbers nobody asked about. Ten is where\nthe caution stops because that is where a live matrix stops being cheap,\nnot because ten runs make a rate trustworthy: eight in ten is still\nconsistent with anything from about half the time to almost always.\n\nSay plainly that the rate alone warrants no edit. An operator who came\nfor a verdict will otherwise fix something, and a skill rewritten against\nfour runs is a skill rewritten against a coin flip. If Step 2 already\nrouted a diagnosed failure inside that same pool, name it here as the\nthing to fix and the rate as the thing to leave alone, so the two do not\nread as one recommendation.\n\n## Step 3: Write it up\n\nLead with the ordering, then one finding per symptom. For each one give\nthe evidence you read, the file, and the edit:\n\n```\nskill_fired: gemini-cli\u002Fgemini-2.5-pro never activated on either run\n(not_triggered: 2, rate: n\u002Fa), so nothing else on that row was measured.\nclaude-code fired on all 9.\n\n1. inputs_resolved 40% on claude-code, missing-owner run 0\n   Evidence  no `match` fired for platform-team; the agent ended on\n             \"who should be listed as the responsible party?\"\n   File      shakedowns\u002Fwrite-plan.cases.toml\n   Edit      the answer's `match` covers \"owner\", not \"responsible\n             party\". Widen the alternation.\n```\n\nTwo habits keep this honest. Quote the report rather than paraphrasing it,\nbecause the distinction between \"asked past your pattern\" and \"guessed\" is\ncarried in the agent's exact words. And when the evidence does not settle\nwhich of two files is at fault, say both and say what would separate them,\nrather than picking the likelier one and presenting it as a finding.\n",{"name":5,"description":7},"main","3ac632ac167bd480d8b07868b7001fc52e2a402c","skills\u002Fanalyze-results",0,"https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown","https:\u002F\u002Fgithub.com\u002FJasperHG90","https:\u002F\u002Fjasperhg90.github.io\u002F","Data Engineer @ Xebia Data. I am interested in data analysis, history, politics, education, policy making and writing. ","2026-09-21 04:18:57",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add JasperHG90\u002Fshakedown --skill analyze-results",{"tool":39,"command":40},"bunx","bunx skills add JasperHG90\u002Fshakedown --skill analyze-results",{"tool":42,"command":43},"pnpm","pnpm dlx skills add JasperHG90\u002Fshakedown --skill analyze-results","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fskills\u002Fanalyze-results\u002FSKILL.md",true,[],[48,55,62,69,76,81],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7AAFD8Q4CRZ5V34GB5655FX","fake-skill","A skill used only by shakedown's own offline tests.","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Ftests\u002Ffake\u002Fskill\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":19},"01M7AAFB03PDQMX0DY841ADDR9","shakedown-canary","Run the shakedown canary check. Use whenever someone asks to run the canary check, verify the harness, or confirm shakedown is wired up correctly.","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fsrc\u002Fshakedown\u002Fcanary\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7AAF55DP6C2XSF7N23STFP3","create-cases","Author a shakedown cases file (`shakedowns\u002F\u003Cslug>.cases.toml`) for an existing agent skill: read the skill, turn what it promises into cases, supply fixtures for the commands it must not really run, and check the result with `shakedown case validate`. Use this whenever someone wants to measure, smoke-test, or add conformance cases for a skill, asks \"how do I test this skill\", points at a SKILL.md and wants cases written, or is editing a `.cases.toml`. Use it even when they only name the skill (\"write cases for connect-agent-to-hub\") without mentioning shakedown or TOML.","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fskills\u002Fcreate-cases\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7AAEMM68JP8A128WHMTJJTX","add-harness","Add a new agent harness (opencode, Gemini CLI, Codex, Amp, a local Ollama-backed runner, or any CLI that loads skills) to a shakedown config so its skill conformance can be measured, and diagnose a harness that `shakedown doctor` reports as not qualifying. Use this whenever someone wants shakedown to run against a harness it does not yet support, asks why a harness fails doctor, asks how to point an existing harness at a different model or provider, or is editing the `[harness.*]` block of a shakedown.toml. Use it even when they only name the tool (\"can this test opencode?\") without mentioning config or TOML.","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fskills\u002Fadd-harness\u002FSKILL.md",[],[],{"skillNo":77,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7AAEEKJAS0KAJRM74K7FEFJ","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fplugins\u002Fopencode\u002Fskills\u002Fcreate-cases\u002FSKILL.md",[],[],{"skillNo":82,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M7AAE6T8RTQBFCD4Q1YAX9EH","https:\u002F\u002Fgithub.com\u002FJasperHG90\u002Fshakedown\u002Fblob\u002Fmain\u002Fplugins\u002Fopencode\u002Fskills\u002Fanalyze-results\u002FSKILL.md",[],[]]