test-evals
Use when writing tests or verifying behavior before code lands — the stage that writes the plan's tests and runs them RED before build; TDD red-green-refactor, the iron law (no production code without a failing test first), seam-first public-interface tests, test sizes, mock discipline. Or when the user says "TDD N loops each batch Opus loop", "batch properly, Opus on each batch before commit", "implement, use TDD", or wants multi-feature TDD with per-batch Opus review before commit. Supports both Hermes-builds and Cursor-builds. Covers pre-commit five-axis review, severity labels, and verifying external review findings before fixing.
- License 许可
- MIT
- 最近更新
- 2026-11-13
- Stars
- ★ 123
- Forks
- 22
01 前言 Front Matter
- name
- test-evals
- metadata
- { "author": "moonlight-lupin", "hermes": { "tags": [ "tdd", "testing", "red-green-refactor", "seams", "test-sizes", "mocks", "batched-tdd", "opus-review", "pre-commit-review", "severity-labels", "test-evals" ], "related_skills": [ "build", "spec", "review-commit", "skill-quality-review", "model-routing", "doubt-driven-development", "hermes-agent-skill-authoring" ] }, "license": "MIT", "version": "2.0.0", "platforms": [ "linux", "macos", "windows" ] }
- description
- Use when writing tests or verifying behavior before code lands — the stage that writes the plan's tests and runs them RED before build; TDD red-green-refactor, the iron law (no production code without a failing test first), seam-first public-interface tests, test sizes, mock discipline. Or when the user says "TDD N loops each batch Opus loop", "batch properly, Opus on each batch before commit", "implement, use TDD", or wants multi-feature TDD with per-batch Opus review before commit. Supports both Hermes-builds and Cursor-builds. Covers pre-commit five-axis review, severity labels, and verifying external review findings before fixing.
02 资源文件 9
- 参考文档references/batched-tdd-opus-loop.md19.0 KB
- 参考文档references/key-rules.md3.4 KB
- 参考文档references/pitfalls-cli-and-env-mechanics.md6.3 KB
- 参考文档references/pitfalls-loop-convergence.md11.6 KB
- 参考文档references/pitfalls-review-findings.md13.0 KB
- 参考文档references/pre-commit-review.md17.0 KB
- 参考文档references/requesting-code-review.md9.2 KB
- 参考文档references/review-report-shape.md6.6 KB
- 参考文档references/test-driven-development.md12.2 KB
03 SKILL.md 正文
Test Evals
Overview
The test-and-evaluation stage of the dev-pipeline: how behavior is pinned before it ships. Test-driven development is the core discipline; the batched multi-feature TDD loop with a per-batch Opus review, the pre-commit verification pipeline, and external-review-finding verification hang off it.
Core principle: If you didn't watch the test fail, you don't know if it tests the right thing.
Tests verify behavior through public interfaces. A bad test is one that breaks on a refactor without a behavior change.
This stage owns the tests
In the pipeline, the plan names each task's test cases (behavior, seam,
expected value and where it comes from). This stage turns them into real
tests, runs each one to a failure for the right reason, and attests the files
(review-commit/scripts/attest_file.py attest) so the build can't quietly
rewrite them. The build implementer makes them GREEN; it adds a test only
for behavior the plan missed, and says so. Output of this stage: the test
files, a RED run log, and one line per test naming the seam note it pins.
When to Use
- User says "TDD N loops each batch Opus loop" or "batch properly, Opus on each batch before commit"
- User says "implement, use TDD" or "TDD, 2 loops each batch, Opus on each batch"
- Writing new features, bug fixes, or behavior changes that need failing tests first
- Reviewing or sizing a change before commit (five-axis review, severity labels)
- Verifying external review findings before fixing them
Don't use for:
- Iterative post-commit review convergence — the
review-commitstage owns that - In-flight doubt about a mid-build step — that's
doubt-driven-development - Judging a skill's own quality — the
skill-quality-reviewefficacy rubric and its Checks stay owned by that skill; this stage links to it, it does not duplicate the rubric
1. Test-Driven Development (TDD)
Core principle: the Overview's — the one stated there is the rule this cycle enforces (see Overview; restated once, applied everywhere in this stage).
RED-GREEN-REFACTOR Cycle
- RED — Write one minimal failing test. Run it. Watch it fail for the right reason (feature missing, not typo).
- GREEN — Write simplest code to pass. Cheating is OK (hardcode, copy-paste). Run test to verify pass.
- REFACTOR — Clean up duplication, improve names, extract helpers — in code THIS cycle wrote or changed, nothing else (surgical changes,
orchestrator/references/coding-principles.md). Keep tests green throughout. - Repeat — Next test for next behavior.
Iron Law
NO PRODUCTION CODE WITHOUT A FAILING TEST FIRST
Code before test? Delete it. Start over. No "keep as reference."
Common Rationalizations to Reject
- "Too simple to test" → Simple code breaks too.
- "I'll test after" → Tests passing immediately prove nothing.
- "Already manually tested" → Ad-hoc ≠ systematic.
- "TDD will slow me down" → TDD is faster than debugging.
Seam first, then size
Pick WHERE a test attaches before deciding how big it is (see Seams and mock discipline below):
- Test at the highest seam that exists — the public API, CLI command, MCP tool, or module's public function the behavior is observed through. Fewer seams are better; one is ideal.
- Go lower only for logic the high seam can't reach cheaply — dense pure-logic edge cases (parsers, money math, date rules) get small tests at their own public function.
- Keep the slow end small — full-system E2E runs are for the few acceptance paths that need the real thing (pre-deploy smoke).
There is no target ratio of unit to integration tests. A seam test that runs against a temp DB is the right test even if that makes "integration" the majority. (v0.1.x carried an 80/15/5 pyramid; it pulled against the seam rule and was dropped.)
Test Sizes
| Size | What it touches | Speed | When to use |
|---|---|---|---|
| Small | One function, no I/O, no network, no DB, no filesystem | Sub-ms | Pure logic, edge cases, regression tests |
| Medium | Multiple functions or real I/O (filesystem, temp DB, localhost network) | Seconds | Integration, API surface, DB roundtrip |
| Large | Full system, external services, real network, multi-machine | Minutes | End-to-end acceptance, pre-deploy smoke |
Label tests by size. If a test is slow, it should be because it's doing real integration — not because it's a unit test with accidental I/O.
DAMP over DRY
Tests favor DAMP (Descriptive And Meaningful Phrases) over DRY (Don't Repeat Yourself). Some duplication in tests is good — each test should read as a standalone narrative. A reader should understand a test without scrolling to shared helpers.
- DRY is for production code — shared logic, one source of truth.
- DAMP is for test code — readability first, some repetition acceptable.
Over-abstracted test helpers (setupGenericFixture(Options)) are worse than 3 tests with copy-pasted setup — the abstraction hides what each test actually tests.
The Beyonce Rule
If you liked it, you should have written a test for it.
Any behavior the system depends on should have a test. "Obvious" behavior that breaks in production was never obvious — it was untested. If a refactor could break it silently, it needs a test.
See references/test-driven-development.md for full methodology.
Seams and mock discipline
A seam is the public boundary where behavior is observed — the place a test attaches to the running system: an API route, an MCP tool, a CLI subcommand, a module's public function.
- Tests verify behavior through public interfaces. A bad test is one that breaks on a refactor without a behavior change. If deleting the test only breaks when internal code is reshaped, the test is implementation-coupled.
- No test at an unconfirmed seam. If you cannot name which public boundary the test observes behavior through, do not write it — it is coupled to internals that can be reshaped without the behavior changing.
For every seam, leave a one-line note on what the test at that seam catches and what it misses (e.g. "the create-order API test catches payload-shape regressions, misses queue-timeout behavior"). The spec stage records these seam notes next to the behavior; the test stage consumes them — every RED test cites the seam note it pins.
Anti-pattern checks — each is a test to rewrite, not a style nit:
- Implementation-coupled: internal-collaborator mocks, or tests against private methods. Rewrite against the public surface.
- Tautological expected values: the test recomputes the truth the same way the code does — it can never fail on a real bug.
- Side-channel verification: asserting through
db.queryinstead of the API. Assert what the caller observes, not where the data happened to land.
Expected values must come from an independent source of truth: a literal, a worked example you did by hand, or the value stated in the spec. Never the same computation the code performs.
One logical assertion per test. "and" in the test name means two tests.
Mocks only at system boundaries: external APIs, time/randomness, filesystem (sometimes); prefer a test DB over a mocked DB. Never mock your own classes or internal collaborators.
Batched TDD + Opus loop
For multi-feature builds the user may structure TDD in batches: features are
grouped into batches, each batch gets N TDD loops (build → verify + fix),
then an actual Opus review before commit — spec → batch features → TDD loops
per batch → Opus review → fix → commit → push → CI. The trigger phrase is
"TDD N loops each batch Opus loop". The full process is kept verbatim in
references/batched-tdd-opus-loop.md, with the rule files below.
Pre-commit review
Before code lands, run the verification pipeline in
references/pre-commit-review.md: five-axis review, severity labels, change
sizing, structural remedies, and the independent-reviewer steps. The
review-commit stage OWNS the convergence loop over review findings
(confirm-round protocol) — this stage runs the pre-commit pass that precedes
it. "If Local Coding CLIs Are Unauthenticated" territory resolves through the
cli-auth-fallback skill — never guess credentials.
Integration Between Patterns
- TDD → Code Review: After TDD implementation, run pre-commit review before push.
- Spike → TDD: Once a spike is VALIDATED, discard the spike code and implement properly with TDD.
- Simplify → Review: Simplify-code is a cleanup pass AFTER implementation. Code review is a quality gate BEFORE commit.
- Debug → TDD: Bug found? Write failing test reproducing it (TDD RED). Fix it (GREEN). Test prevents regression.
Red Flags
- Writing production code without a failing test first
- "It works, that's good enough" — working but unreadable/insecure/architecturally wrong creates compounding debt
- "I wrote it, so I know it's correct" — authors are blind to their own assumptions
- "We'll clean it up later" — later never comes; the review is the quality gate
- "AI-generated code is probably fine" — AI code needs MORE scrutiny, not less; it's confident and plausible even when wrong
- "The tests pass, so it's good" — tests are necessary but not sufficient; they don't catch architecture, security, or readability issues
- "The refactor makes it cleaner" — relocating complexity isn't reducing it; count the concepts a reader must hold
- A refactor that moves code around without making branches disappear
- A change that grows an already-large file instead of decomposing it
- New conditionals scattered into unrelated code paths (a missing abstraction)
- A near-duplicate of an existing canonical helper
- Review comments without severity labels — makes it unclear what's required vs optional
- Accepting "I'll fix it later" — it never happens
- A bulk "bump dependencies" PR with no changelog review
- Spiking without a verdict (spike code left in the codebase)
- Trusting the builder's self-report without independent verification
- Hand-coding fixes when the review loop is the active workflow — non-trivial code changes are delegated to the builder lane; hand-editing is only acceptable for trivial mechanical fixes, and say so explicitly when deviating (the
review-commitstage's delegate rule)
Verification
After completing any development workflow task, confirm:
- [ ] Tests written before production code (TDD) — or explicit justification for skipping
- [ ] Every test attaches at a named seam (the highest that reaches the behavior) and cites its seam note
- [ ] Independent review completed (not self-verified) — another agent or model
- [ ] Findings labeled with severity (Critical / Required / Optional / Nit / FYI)
- [ ] Change size within target (~100 lines good, ~300 acceptable, ~1000 split)
- [ ] No dead code left behind (or explicitly listed for future removal)
- [ ] Refactoring separated from feature work (or explicitly combined at reviewer discretion)
- [ ] Simplification reduced concepts, not relocated them
- [ ] Spike code discarded or explicitly marked for reuse
- [ ] External review findings verified independently before fixing
Stage-exit confirmation
After this stage, the TDD verification checklist above must pass and every rule file listed in References must be reachable from this file.
References
references/batched-tdd-opus-loop.md— verbatim batched TDD + Opus review process ("TDD N loops each batch Opus loop"); §6 points at the review-commit stage's confirm-round-protocolreferences/test-driven-development.md— full TDD methodology: iron law, cycle detail, rationalization tables, red flags, bug-fix example, Hermes integration, testing anti-patternsreferences/pre-commit-review.md— pre-commit verification pipeline: Five-Axis Review, Severity Labels, Change Sizing, Structural Remedies, Steps, Read-Only Codebase Audits, external-review verificationreferences/requesting-code-review.md— pre-commit code verification dimensions (also used by the build stage's final integration review)references/pitfalls-loop-convergence.md— loop convergence: public-interface tests, RED delegation, pins, reachability, downstream reviewreferences/pitfalls-cli-and-env-mechanics.md— CLI, env, dispatch timing, turn caps, gh, pre-existing CIreferences/pitfalls-review-findings.md— review findings: warnings, payloads, provenance, injection, concurrency, timestampsreferences/review-report-shape.md— report shape to demand in every review brief; per-batch review session records stay in the source archivereferences/key-rules.md— read before the first batch: RED before GREEN, review-before-commit, specificgit add, CI after push, independent verification, BLE001
Where it fits
Test-evals is stage 3 of the dev-pipeline pipeline: grill → spec →
test-evals → build → review-commit. Tests are written in this
stage; the build stage runs the implementation (ticket title restated before
starting, dependency injection for mockability). Each seam note written at
the spec stage is consumed here at test-writing time.
04 同仓库更多 Skills
website-scraping
Generic playbook for extracting structured data from any website — hotel prices, flight fares, e-commerce listings, real estate, jobs, competitor product catalogues, anything where the goal is to turn one or more URLs into clean records on disk. Use this skill whenever the user mentions scraping, harvesting, extracting, or pulling data from a website; names a specific site or competitor they want data from; asks how to handle JavaScript-rendered pages, Cloudflare blocks, bot detection, Turnstile challenges, or Playwright; wants to monitor prices over time; needs to automate a copy-paste research task; or says things like "just get the data from X" or "I need a script that grabs N from site Y". Covers recon, picking the lightest extraction tool that works, surviving anti-bot defences, and writing clean JSONL output with a run manifest. Scope is scraping only — landing data in a database, scheduling, and downstream pipeline work are explicitly out of scope and live in other tools.
greet
Greets a named person warmly.
skill-quality-review
Use when auditing or reviewing skill quality — batch checks across a library, deep per-skill grep methodology, usage stats, efficacy testing, or fix planning. Single consolidated home for the writing-for-agents review framework and the 3-surface review methodology.
people-enrichment
Use when the user has a list of names or companies and wants their job, employer, work history, LinkedIn URLs or company profiles looked up, or wants to find people or companies matching criteria — trigger on phrases like "enrich these contacts", "find their LinkedIn", "look up these people", "who works at X", "find directors at Y", "enrich these companies", or "get firmographics for". Enrich and search People Data Labs (PDL) Person and Company data, writing results to .xlsx. Five operations: enrich named PEOPLE into a profile, employment history and LinkedIn URL; IDENTIFY an ambiguous person as several scored candidates; SEARCH people by criteria (company, title, location); enrich COMPANIES into firmographics (industry, size, employees, HQ, founded, LinkedIn); and SEARCH companies by criteria. Do not scrape LinkedIn directly — this uses a licensed data aggregator (PDL) instead, for legal and reliability reasons.
notebooklm-mode
Source-grounded research pipeline inspired by NotebookLM. Collect verbatim source extracts into a temporary workspace, then answer questions and create deliverables grounded in those sources. Two grounding modes: strict (vault-only) and augmented (vault + labeled [background] knowledge). Trigger when the user asks to research a topic from sources, wants NotebookLM-style grounded answers, asks to build a resource vault, or says "notebooklm mode".
news-monitoring
Use when a topic or news stream needs recurring monitoring — cover it with web search, multi-language sources, digest formatting, and automated delivery via Hermes cron jobs. Covers search strategy, source selection, Chinese-language platforms, and digest templates.