eval-ladder
Design and audit the eval ladder for an agent system: build tiers bottom-up from observed failures, pick the cheapest rung that can catch a given regression, name what each rung structurally cannot prove, validate every LLM judge against human labels (TPR and TNR separately, never raw agreement), and score irreversible-action scenarios pass^k rather than by majority. Use when designing, auditing, or defending a test/eval strategy for an agent, skill, or prompt; when adding an eval tier or LLM judge; when a suite is all-green and you cannot say what it would catch; or on phrases like "assess our testing strategy", "do we have eval coverage", "is this judge trustworthy", "pass@k or pass^k", "what does this tier actually prove".
- License 许可
- MIT
- 最近更新
- 2026-11-12
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- eval-ladder
- license
- MIT
- description
- Design and audit the eval ladder for an agent system: build tiers bottom-up from observed failures, pick the cheapest rung that can catch a given regression, name what each rung structurally cannot prove, validate every LLM judge against human labels (TPR and TNR separately, never raw agreement), and score irreversible-action scenarios pass^k rather than by majority. Use when designing, auditing, or defending a test/eval strategy for an agent, skill, or prompt; when adding an eval tier or LLM judge; when a suite is all-green and you cannot say what it would catch; or on phrases like "assess our testing strategy", "do we have eval coverage", "is this judge trustworthy", "pass@k or pass^k", "what does this tier actually prove".
- compatibility
- PORTABILITY: pure analytical and authoring discipline — no hooks, no subagent-spawning tool, no harness-specific primitive. The rungs are ordinary scripts, config files and prose that any harness can run; the procedure is reading and reasoning. Works unchanged on claude-code, codex, gemini, cursor.
02 资源文件 5
- 参考文档references/eval-surfaces.md3.7 KB
- 参考文档references/integrity-hazards.md4.9 KB
- 参考文档references/judge-alignment.md4.2 KB
- 参考文档references/metric-choice.md5.6 KB
- 参考文档references/source-map.md8.2 KB
03 SKILL.md 正文
eval-ladder
Invariant
Never present an eval result as evidence beyond what its tier structurally proves: every green is reported with its blind spot, every LLM-judge verdict is bounded by that judge's measured TPR/TNR against human labels, and a scenario guarding an irreversible action passes only when EVERY trial passes (pass^k) — never on a k-of-N majority.
A green check is a claim. The claim is not "the system works"; it is "this tier's specific probe did not fire." Stating the difference is the whole skill.
When to use this
- Designing an eval suite for an agent, skill, prompt, or tool from scratch.
- Auditing an existing suite: what does it cover, what can it structurally never cover, where would a real regression walk through.
- Adding a tier, a judge, or a scenario — deciding which rung it belongs on.
- A suite that is all-green and has never gone red: it may be measuring nothing.
- Reviewing someone's claim that a change is "verified" or "tested".
- Choosing a metric: pass@k, pass^k, a pass-rate floor, or a hard gate.
Do not use it to write the eval's actual content for a domain you have not looked at. The first rung is looking at real failures; this skill will send you there rather than let you skip it.
The order that matters most
Build bottom-up from observed failures, not top-down from imagined invariants. The single highest-return activity is error analysis: sample 20–100 real traces, have one person write a free-text note on the first thing that goes wrong in each, cluster those notes into 4–8 failure modes, then build one narrow binary check per mode. Suites authored from design intent test the failures you imagined; suites authored from traces test the ones you have.
A suite with no error-analysis origin is a hypothesis, not a measurement. Say so when you report it. See judge-alignment.md.
The ladder
Cheapest first. Each rung catches a class of defect the rungs below it structurally cannot — that "cannot" is what earns the rung its cost.
| # | Rung | Catches | Structurally cannot |
|---|---|---|---|
| 0 | Structural — parses, wiring, manifests, load-bearing greps | Broken plumbing, deleted invariant text | Whether a sentence still means anything |
| 1 | Discriminating corpus — mutate a known-good baseline, assert rejection for the right reason | A gate that has stopped gating | Defects it has no fixture for |
| 2 | Code assertion — regex, schema, state query, per observed failure | Anything a deterministic predicate can decide | Subjective quality; unanticipated shapes |
| 3 | LLM judge — binary, few-shot-critique-grounded rubric | Residual subjective failures | Anything beyond its measured TPR/TNR |
| 4 | Trajectory / decision point — the path, or the next move at a frozen prefix | Wrong route, wrong tool, wrong order | That the chosen path then worked |
| 5 | Outcome / environment state — diff the world before vs after | "It said done but nothing happened" | Intent, and anything outside the diffed scope |
| 6 | Sandboxed cross-harness end-to-end | Harness-dependent behavior; the real loop | Non-safety qualities; costs real money |
| 7 | Human demonstration — the thing run on real material, misses included | Whether the change is worth having | That one run generalizes — no repeatability, no unseen inputs |
Two rules govern movement on the ladder:
- Descend before you ascend. If a deterministic predicate can decide a failure, never buy a judge for it. Judges cost 100+ labels, ongoing validation, and a permanent uncertainty band.
- Grade the surface closest to the harm. Text output is the weakest surface: an agent can say "your flight is booked" with no row in the database. Prefer environment state, then trace, then output. See eval-surfaces.md.
The audit procedure
For each tier in the suite under review, answer all five. A tier that cannot answer #2 or #5 is decoration.
- What does it prove? One sentence, in terms of a defect it would catch.
- What can it structurally not prove? If this is blank, the tier is being over-claimed. Write it down and publish it beside the green.
- What is the grader, and is the grader validated? Deterministic, or a model? If a model: TPR and TNR against held-out human labels, or it is an opinion. See judge-alignment.md.
- What is the metric, and does it match the question? Capability → pass@k. Reliability, and anything irreversible → pass^k. When the change moves work rather than removing it, check that the metric counts both sides: a number measuring only the arm that got smaller reports a relocation as a saving. See metric-choice.md.
- How would it go red? Name a concrete change that turns it red. If you cannot, it is not a gate. Then check the twin: is there a must-not-fire control proving the green is not what the bare system does anyway?
Run the harness; do not read it. Question #5 is answered by executing the tier and counting what it reported, not by reading its source or its comments. A runner's description of itself is a claim like any other, and the gap between "defines the same helpers" and defining the same helpers is invisible on the page and obvious in one run. Compare the count of checks a tier reports against the count it contains, and treat any silent difference as the finding.
Then sweep the whole suite for the failure modes that live between tiers rather than inside one — saturation, tuning on the gate, criteria drift, the harness confound, contamination. See integrity-hazards.md.
Reporting
Report coverage as a matrix of surface × rung, not as a count of tests. State
the blind spots as first-class findings. Rank recommendations by
frequency × severity, and say which rung each belongs on and what it costs.
Never write "fully tested". Write what was probed, by what grader, at what confidence, and what remains unprobed.
References
- eval-surfaces.md — the five gradable surfaces, which rung reaches each, and why output-only grading is the weakest.
- judge-alignment.md — error analysis, binary judges, critique shadowing, and validating a judge with TPR/TNR.
- metric-choice.md — pass@k vs pass^k, floors, FAULT-vs-FAIL separation, difficulty calibration, and starvation.
- integrity-hazards.md — saturation, tuning on the gate, criteria drift, the harness confound, contamination, judge bias.
- source-map.md — which primary source each rule comes from, for when a claim here needs checking at the source.
04 同仓库更多 Skills
wayfinder
Chart a multi-session effort as a labeled map of typed decision tickets (grilling / prototype / research / task) with explicit dependencies and an open frontier agents self-assign into. Plans; never executes. Use this skill whenever the user is working with wayfinder, and on phrases like "chart this effort", "break this into tickets", "what's the frontier", "plan across sessions", or "map out the dependencies for this work".
wave-research
Staged, adversarial research that tries to break a thesis before believing it: waves, checkpoints, falsifiable sub-claims, rival hypotheses, an ACH matrix, source-incentive tags and a verified final thesis. Also carries Workflow templates for a one-pass research-and-verify fan-out. Use to stress-test or vet a contested claim; not for quick lookups.
second-opinion
Offer-only validation pipeline for a verdict or recommendation. Use when the user asks to validate, verify, fact-check, or get a second opinion on a claim or recommendation — "are you sure", "double-check this", "second opinion". Batches fact-checking against sources, adds scoped advisor personas for the angles that need judgment, then re-emits the response grouped into verified, flagged, and conflict, with a mandatory validation delta. Requires a subagent-spawning tool; without one it must decline rather than simulate.
machine-voice
Compression rules for standalone machine-read artifacts the reader scans — agent traces, tool-call logs, status lines, progress updates, handoff notes, state dumps, config, schemas, reference cards, and structured data another agent parses. Applies three layers: lexical compression, markdown structure as navigation, and emojis as typed status markers. Reach for it when emitting one of those artifacts on its own. Companion to human-voice, which governs prose a human reads and also owns any table or block that serves surrounding explanation. Neither skill applies to code or file contents the user will save, commit messages, creative writing, or turns that are only a clarifying question — those ship unstyled and uncompressed.
human-voice
Style for prose a human reads — answers, recommendations, explanations, research summaries, comparisons. Use on every conversational reply that contains prose for a person to read: triage intent, lead with the verdict, keep it scannable, tag claim confidence. Reach for it whenever answering a technical question, comparing options, or making a recommendation, even when the user says nothing about style. Companion to machine-voice, which instead governs standalone machine-read artifacts — agent traces, tool-call logs, status lines, progress updates, handoff notes, state dumps, config, schemas, reference cards, and structured data another agent parses. Neither skill applies to code or file contents, commit messages, creative writing, or turns that are only a clarifying question — those ship unstyled.
ai-writing-mistakes
Catch and repair the tells that mark prose as machine-written — the negation-flip ("it's not X, it's Y"), rule-of-three padding, hedge stacking, unearned intensifiers, filler verbs like delve and leverage, participle trailers ("...ensuring seamless integration"), sycophantic openers, empty closing summaries, and uniform em-dash/sentence rhythm. Use when drafting or revising any prose a human will read, including documents you were asked to author — READMEs, docs, PR bodies, release notes, emails — and when the user asks whether something "sounds like AI" or wants writing tightened. This is a wording pass, not a voice: human-voice still decides layout and confidence tags, machine-voice still owns traces and logs, and neither this skill nor those two touch code, commit messages, or quoted text.