nacharbeit-review
Use to run the calibrated judgement review of one or more Claude Code plugins against the nacharbeit rubric: a sonnet finder tuned on planted-defect fixtures until it clears a recall floor for every rule family, measured once on a sealed hold-out, then two-lens finders with gap rounds per (plugin, kind) batch, a haiku routing simulation for cannibalization, an adversarial refuter, and an opus synthesis into a tiered backlog and a findings report. Trigger on 'review this plugin against the Anthropic standard', 'where do these skills and hooks fall short', 'run the nacharbeit review', 'prompt-quality review'. Runs nacharbeit-lint first; never applies a fix (that is nacharbeit-fix). Writes only analysis/nacharbeit/ and, inside werkstoff, the generated findings report.
- License 许可
- MIT
- 最近更新
- 2026-11-02
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- nacharbeit-review
- description
- Use to run the calibrated judgement review of one or more Claude Code plugins against the nacharbeit rubric: a sonnet finder tuned on planted-defect fixtures until it clears a recall floor for every rule family, measured once on a sealed hold-out, then two-lens finders with gap rounds per (plugin, kind) batch, a haiku routing simulation for cannibalization, an adversarial refuter, and an opus synthesis into a tiered backlog and a findings report. Trigger on 'review this plugin against the Anthropic standard', 'where do these skills and hooks fall short', 'run the nacharbeit review', 'prompt-quality review'. Runs nacharbeit-lint first; never applies a fix (that is nacharbeit-fix). Writes only analysis/nacharbeit/ and, inside werkstoff, the generated findings report.
03 SKILL.md 正文
Measure before asserting. The finder that grades real files is the one that first passed a planted-defect calibration for every family it will grade; a finder that has not is a reviewer with an unknown false-negative rate, and its output is an opinion.
Steps
-
Preflight. Run
nacharbeit-preflight. Stop on an open fix lock. Note every skipped checker; the report will carry the same gaps. -
Build and bake the args (this also runs the lint; every path is a flag with a default —
--helplists them. The calibration fixtures ship inside the plugin and are staged intoanalysis/nacharbeit/fixtures/, so this works in any repository):uv run --script "${CLAUDE_PLUGIN_ROOT}/scripts/build_args.py" [--plugin <name> ...] [--docs-root docs]It refuses when a batch kind has no tuning + sealed fixture pair under the fixtures root (
--fixtures-root DIRswaps the set): a finder cannot grade what it was never calibrated on. It prints the batch list; tell the user the plugin count, the batch count and the fixture count before launching anything.uv run --scriptreads the script's PEP 723 header and suppliespyyaml. Withoutuv, run the same path withpython3 -B; that needspyyamlimportable, and the linter stops with exit 2 and says so rather than grading frontmatter it cannot parse. -
Launch the review with the Workflow tool, by the baked script path and never with hand-built args — the args are 170+ KB and the leak assertions live in them:
Workflow({ scriptPath: "analysis/nacharbeit/run.js" })The workflow calibrates (up to three rounds, an opus rewriter between them), measures the sealed hold-out, refuses to grade if any family falls below its floor, then finds, routes, judges, verifies and synthesizes. Expect on the order of forty agent calls per plugin. A return with
completed: falsenames the calls that returned nothing; resume it withresumeFromRunId, never persist it.Without the Workflow tool: dispatch the
component-finderagent once per batch and lens from the printed batch list, collect its findings, and label every one uncalibrated in the report. Do not runwrite_results.pyon them — it refuses a return value without a calibration block, and so does the fix pass. An uncalibrated review is a reading, not a measurement, and is presented as one. -
Persist, or refuse. Save the workflow's return value to
analysis/nacharbeit/return.json, then:python3 -B "${CLAUDE_PLUGIN_ROOT}/scripts/write_results.py" analysis/nacharbeit/return.jsonIt writes
run.jsonlast, so its presence means the other three are complete. A refusal leaves aFAILED-<stamp>marker and nothing else; report the reason and stop — a stalerun.jsonunder a fresh marker is exactly the statebuild_report.pyrefuses to render. -
Render the report and present it:
python3 -B "${CLAUDE_PLUGIN_ROOT}/scripts/build_report.py"Inside werkstoff this writes
docs/prompt-quality-findings.md; elsewhere pass--outand--no-vitepress. Lead with section 1 (how much to trust the rest: recall per family, sealed recall, router accuracy), then the per-plugin verdicts, then the backlog by tier with every opus- and human-tier entry named — those are the ones a person must decide.
Output format
nacharbeit review 2026-09-09T06:12:04Z · rubric 3f9c1a2b7d0e · 3 plugins, 19 batches, 20 fixtures (kinds: assets, docs, hooks, manifest, scripts, skills)
calibration: r1 recall 0.76 (min angle 0.40, min family 0.50) → rewrite → r2 recall 0.83 (min angle 0.60, min family 0.67)
sealed hold-out: 0.79 overall; Q 0.81, HQ 0.75, SQ 0.67, AQ 0.75, PQ 0.80, DQ 1.00 — every family above its floor, finder frozen
router proxy: 0.97 top-1 on 70 known-answer prompts → collisions are measured (24 pairs judged); routing handed to the finders
routing findings: 3 Q-ROUTE-MISS, 1 Q-CANN-CAPTURE (produced in code from the simulation)
raw 312 → verified 164 (+9 from the critic round); declined 5 (fix would make it worse); lint 41
uncalibrated kinds: agents, commands, workflows — their findings carry calibrated:false
plugin verdict verified lint summary
andon needs-work 38 9 andon-loop's argument-hint promises a filter no phase reads; …
takt sound 2 3 …
lehre needs-work 57 8 …
backlog: haiku 31 · sonnet 96 · opus 2 · human 4
held for a person (never auto-applied):
[opus ] zirkel: plugins/zirkel/workflows/solve.js — design a real gate for each stage's output_contract …
[human] andon: plugins/andon/skills/andon-loop/SKILL.md — decide whether andon-loop supports scoped runs …
report: docs/prompt-quality-findings.md (682 lines)
next: nacharbeit-fix applies the haiku and sonnet tiers under the lock; the four held entries need you.
Rules
- Never retune the finder by hand, and never edit a fixture after seeing what the finder missed on it. Fix the rubric or add a fixture, then rerun from step 2.
- Never pass
argsinline. The bakedrun.jsis the only launch path. - A
completed: falsereturn is resumed, not persisted.write_results.pywould refuse it anyway; do not work around the refusal. - Never grade a kind that has no sealed fixture.
build_args.pyrefuses; do not remove the fixture requirement to make it pass. - Read the transcript of any single-pass surprise before believing a tally.
- Routing below 0.8 is hints, and says so. The review then emits no
Q-ROUTE-MISSorQ-CANN-CAPTUREand records them underrouting.rulesSkipped; report that, never a rate. - A
Q-ROUTE-MISSfix is verified by re-measurement in a fresh process, never by re-reading the description: definitions load once per session. - Declined findings are not backlog. Report their count; never hand them to
nacharbeit-fix. calibrated: falseis said aloud. A finding from a kind with no fixture pair of its own is reported as uncalibrated, not merged into the calibrated tally.
Resources
scripts/build_args.py,scripts/write_results.py,scripts/build_report.py— run, in that order.workflows/review.js— read to understand the phases; never edit it mid-run (its baked copy is what runs).agents/component-finder.md— the fallback finder for sessions without the Workflow tool.references/rubric.md— the standard every finding cites; read a rule before disputing a finding.scripts/ambiguous-prompts.json— werkstoff's hand-written ambiguous prompts for the routing simulation; another repository passes its own with--ambiguous.scripts/fixtures/calibration/— the planted-defect fixtures the finder is calibrated on; never edited after a finder has seen them.
04 同仓库更多 Skills
toy-check
A skill that checks things in the repository and reports what it finds.
demo-check
Reports which generated files changed since the last generation run. Use when the user asks what drifted. Read-only; never edits a file.
demo-check
Reports which generated files changed since the last generation run. Use when the user asks what drifted. Read-only; never edits a file.
demo-check
Reports which generated files changed since the last generation run. Use when the user asks what drifted. Read-only; never edits a file.
demo-check
Reports which generated files changed since the last generation run. Use when the user asks what drifted. Read-only; never edits a file.
zirkel-verify-assumptions
Checks exactly ONE named assumption against real evidence in at most 3 steps and reports one outcome: confidence raised (>=90), reading changed, or still unresolved. Use to resolve a single load-bearing uncertainty — typically a blocking entry surfaced by zirkel-clarify-scope: "verify this one assumption", "is it actually true that X", "check this before we rely on it". For multiple uncertainties, invoke once per uncertainty — the step budget is never shared.