AGENT SKILL

eval-run

Execute a project's eval and produce a results file - locate the golden set, scaffold a runner for the project's stack when none exists, grade every case by its own rule, and write per-case results plus a headline computed from that file. Reports verdict-first against the spec's target and never edits a golden case to raise a score. Use when the user asks to run the eval, score the golden set, or produce eval results.

License 许可
MIT
最近更新
2026-09-28
Stars
★ 2
Forks
0

01 前言 Front Matter

name
eval-run
description
Execute a project's eval and produce a results file - locate the golden set, scaffold a runner for the project's stack when none exists, grade every case by its own rule, and write per-case results plus a headline computed from that file. Reports verdict-first against the spec's target and never edits a golden case to raise a score. Use when the user asks to run the eval, score the golden set, or produce eval results.
argument-hint
[eval-dir | notes]

02 资源文件 2

03 SKILL.md 正文

/eval-run — execute the eval, honestly

/eval-spec writes the target before the code exists; /audit eval reviews results after; /ship's gate 3 compares a headline to a target. All three assumed a runner nobody produced. This skill is that runner — the step that turns "the eval is the spec" from a claim into a loop.

Running an eval costs money and time when the system under test calls a model API. State the case count and that cost before executing, then run — the user asked for a run, so this is not a permission request (conduct rule 5), it is a disclosure.

Adjacent skills: /eval-spec (writes the spec and golden set before any code; /eval-run executes it) · /audit eval (audits the results this produces, including recomputing the headline) · /ship (gate 3 compares this headline against the spec's target).

<!-- acstack:runtime --> Run before the skill's steps — per invocation, not per session (4.36); failures degrade to markdown:

link="$(readlink "$HOME/.claude/skills/health" 2>/dev/null || true)"   # empty = not symlinked
pack="$(dirname "$(dirname "$link")")"   # NEVER trust this unless $link was non-empty
if [ "${link#/}" != "$link" ] && [ -x "$pack/bin/acstack-config" ] && ! "$pack/bin/acstack-config" runtime | grep -q '=off'; then
  "$pack/bin/acstack-config" || true          # resolved keys, with sources
  "$pack/bin/acstack-update-check" || true    # ≤1 fetch/day; silent ONLY if already checked today
  "$pack/bin/acstack-recall" || true          # LEARNINGS.md + bug-class names, capped 3KB
else
  echo "runtime off — proceeding without recall/update-check"
fi

<!-- /acstack:runtime -->

<!-- acstack:principles -->

Operating principles

  • Be direct. Push back in writing when the plan or the user is wrong. No sycophancy.
  • Never delete a decision. Supersede it: ~~old~~ → **Verdict (YYYY-MM-DD):** new call — reason.
  • Never fix, tune, or delete a test or eval case to raise a score. Log the miss honestly and leave the case unchanged.
  • Name exact things: regex patterns, function signatures, model names, before → after numbers. Never "fixed bugs".
  • Attribution: follow the project's attribution setting (default none) — no AI-tool mentions in generated docs, no attribution trailers in commits or PRs. Commit with explicit -m/-F messages only.
  • Config: read .claude/acstack.md at the project root (fall back to ~/.claude/acstack.md) before acting. ## Settings keys override pack defaults; a ## <skill-name> section overrides both. Unknown keys and sections are ignored.
  • Docs: BRIEF.md (frozen seed) / PLAN.md (living plan) / JOURNAL.md (rolling journal). If the repo uses legacy names (PLANNING_PROMPT.md / PLANNING.md / STATUS.md), use those instead — never create both.
  • Recall: if LEARNINGS.md exists at the project root, read it before starting.
  • Conduct: follow the acstack-conduct block in this repo's AGENTS.md — the word is the mode; the user sets the pace.
  • Hackathon lane: if the project's AGENTS.md carries the acstack:hackathon-lane block, only /do changes the repository during the event. Any other skill that would write a tracked file, commit or push says what it would have done and stops; a change that is not a task goes through the lane's operator route. <!-- /acstack:principles -->

One document set. Resolve exactly ONE BRIEF/PLAN/JOURNAL set and name its path in the report's scope line. If more than one candidate set exists — a monorepo, nested products, an apps/* tree each with its own docs — list the candidates and STOP. Never pick one silently: a confident answer about the wrong product is worse than no answer (conduct rule 8).

The sequence

  1. Locate the eval. eval/spec.md and eval/golden.jsonl (or the directory named in the argument). Either missing → say WHICH is missing, point at /eval-spec, and stop. Never scaffold a golden set here; writing the target is a deliberate act that belongs to /eval-spec, and inventing cases at run time is the never-inflate rule's exact failure mode.

  2. Prefer the runner that exists. If spec.md's run command names a runner present in the repo, run THAT and skip step 3 — an existing eval artifact is never regenerated (/eval-spec's step-0 rule).

  3. Scaffold only when there is none. Detect the stack: package.json → eval/run.mjs; pyproject.toml or requirements.txt → eval/run.py; neither, or both → stop and ask. Never guess a stack. Build from references/runner-template.md.

    A run command in the spec settles the stack — it is not a guess. If spec.md names eval/run.py, the language is already decided by the committed spec, and stopping to ask which stack to use is a deadlock over a question already answered. Take the extension from the spec's run command and say that is where it came from. Only a spec that names no runner leaves the question genuinely open.

    The runner needs one fact the spec often lacks: how to invoke the system under test — a CLI command, an HTTP endpoint, or a function import. If spec.md does not say, ask, then record the answer in spec.md's run section as a dated addition (that file is living; the golden set is not).

    Unattended, the same settled-not-guessed rule applies: when nobody can answer, a committed document that states the exact invocation — a PLAN exit criterion, an acceptance line — settles the question; record it as the dated addition with the source named. Only when no committed document states the invocation does an unattended run stop and say what is missing. Never derive it from reading the subject's code: the code cannot distinguish its intended interface from an accident, so that is a guess wearing evidence.

    The invocation carries isolation flags and a pinned subject model, or it is not measuring the subject. Without isolation the operator's user-level skills, hooks, memory and CLAUDE.md discovery, and output styles all ride into the run — including into the BASELINE arm of an A/B, where they can make a candidate look better or worse than it is. Without a pin the run silently uses whatever the operator or the CLI release defaults to, so two runs are not comparable. Flags per stack are in references/runner-template.md; state what each drops AND what still leaks. An unpinned model STOPS the run — see the template's SUBJECT_MODEL guard.

    Then the non-regression gate. references/regression-gate.py compares the new results file against the LAST COMMITTED one, per category, and blocks when any category falls. The headline cannot see this: a change that lifts the overall percentage while breaking every refusal case passes both /ship's target gate and the spec's category minimums, since minimums constrain the golden set's composition, not one run against another. No committed baseline PASSES and says so — never silently, because an absent baseline read as a clean pass is the false confidence the gate exists to remove.

  4. Run it, then read the file. The runner writes eval/results/<UTC-timestamp>.jsonl. The headline is computed from that file — overall %, per-category %, refusal % — and never assembled by hand from what the run appeared to do. If the two ever disagree, the file wins and the discrepancy is the finding.

  5. Report (shape below). Propose PLAN edits; apply none.

Grading rules

Per case, from its grade_rule, aligned with the canonical grader rules in ../eval-spec/references/grader-rules.md and with /audit eval:

  • exact — normalized equality: trim, collapse whitespace, NFKC, and fold case — unless the case carries case_sensitive: true, set when the output's shape is itself part of the contract; then case is kept and a right answer in the wrong case fails on shape. Same answer, not same keystrokes.
  • concept — the expected concept is present. The scaffold implements this as normalized substring containment, which is the FLOOR, not the ideal: it is literal enough to produce brittleness. When a case fails here and the answer was right, widen the grader — never touch the case (/audit eval's "grader brittleness" bucket).
  • numeric-tolerance:<x> — parse both sides, compare within ±x absolute, or ±x% relative to the expected value when the suffix is % (both forms are documented by /eval-spec's template).
  • rubric:<name> — not machine-gradeable. The runner records the answer, marks the case needs-rubric-review, and excludes it from the headline while naming it. A human or a judge model scores the rubric's named dimensions; a rubric with unnamed dimensions is a spec defect, not a pass.

Denominator discipline, because a headline is only as honest as what it counts: cases marked "status": "needs-data" are skipped and reported as skipped, never counted as passes; "status": "superseded" cases are excluded entirely; rubric: cases are excluded from the headline because no machine graded them, and are reported for review; acceptable_failure applies only when the case carries a written reason (in either shape it is written — a bool with a sibling reason, or an object), never to an ungraded case, and every application is listed.

Every exclusion is named in the report. A case that leaves the denominator quietly moves no percentage at all, so it cannot be caught by looking at the number — which makes silent exclusion the most dangerous of the false-pass family.

Hard rules

  • Never edit golden.jsonl to raise a score. A genuinely wrong case is superseded through /eval-spec, with a new id and a written reason. Editing a case to pass is the one thing this pack refuses outright.
  • Never hand-transcribe the headline. It comes from the results file, every time.
  • A run that errors part-way reports what completed and what did not — a partial run is never presented as a full score.
  • A subject that cannot be invoked yields NO SCORE, not 0%. When the system under test needs a credential or endpoint the environment lacks, record honest per-case errors in the results file, report the headline as measuring the environment rather than the subject, and name what is missing in Scope. Never set, invent, or mock the credential to make the run go — a mocked subject scores the mock.
  • The results file is committed with the run: an eval score whose evidence is not in the repo is an assertion.

Report shape

Verdict first — <headline>% vs target <n>% plus MEETS TARGET or BELOW TARGET by <x>; when nothing was graded, the verdict is instead NO SCORE — <what is missing> (per the hard rule above — a zero-graded run has no headline to compare) — then:

  • per-category table (category | cases | passed | % | target)
  • failures, grouped by the /audit eval buckets (prompt issue, grader brittleness, provider flake, data issue, parser issue, genuinely ambiguous) — classification proposed, since confirming it is /audit's job
  • acceptable_failure applications, each with its reason
  • skipped cases and why
  • the results file path, and the exact command that produced it
  • Scope: what was NOT run (needs-data cases, categories with no cases, an unreachable system under test)

Positive control

A golden set containing one deliberately failing case MUST produce a sub-100% headline. A runner that reports 100% by construction — because it counts skips as passes, swallows errors, or compares nothing — is the false-pass class this pack exists to catch, and it is indetectable from a green report. fixtures/eval-run/ seeds exactly that case.

04 同仓库更多 Skills

AaravChadha
AaravChadha/acstack

why

Decision archaeology - answers why the code, config, or plan is the way it is by searching the record in a fixed order: BRIEF constraints, dated PLAN decisions, JOURNAL entries, then git history. Stops at the first real answer, cites it with a date, and says no recorded rationale rather than inventing one. Use when someone asks why something is this way, why a choice was made, or where a decision is written down.

★ 2
AaravChadha
AaravChadha/acstack

verify

Audit a completion claim made by someone else - another session, agent, or teammate - against the running system rather than the diff. Re-derives what the claim's acceptance demands, runs it, and returns CONFIRMED, OVERSTATED, FALSE or UNVERIFIABLE with the command and its output. Never fixes, never edits the claim or the acceptance. Use when handed a "done", "all green", "merged" or "it works" you did not produce yourself.

★ 2
AaravChadha
AaravChadha/acstack

triage

Backlog hygiene sweep - stale items, duplicate pairs, missing acceptance criteria, unblocked-but-unassigned work, and milestone burn, delivered as a report first with only user-approved actions applied. Works on GitHub Issues in tickets mode and on PLAN.md checkboxes in document mode. Use when the user asks to triage, groom, or clean up the backlog or the plan's open tasks.

★ 2
AaravChadha
AaravChadha/acstack

ticket

Capture a brain-dump as a well-formed work item - verb-first title, acceptance criteria, file paths when known, an out-of-scope line. Files a GitHub issue in tickets mode or appends a numbered PLAN.md task in document mode; unknowns are marked TBD, never invented. Use when the user brain-dumps an idea, bug, or task to capture for later, or asks to ticket, file, or note something as work.

★ 2
AaravChadha
AaravChadha/acstack

ship

Branch-level release with five gates before the act - clean-state, tests, eval-vs-target, docs drift, and attribution sweep; any failing gate stops the release with its output. Then push, and under push: branch-pr open a report-shaped PR wiring the issue-closing reference in tickets mode or PLAN task IDs in document mode. Use when the user asks to ship, release, cut, or open the PR for a feature or branch.

★ 2
AaravChadha
AaravChadha/acstack

secure

Security review with a confidence gate - a finding exists only with a concrete exploit scenario and a high/medium/low confidence rating; everything else goes to a worth-hardening list, never inflated. Sweeps five surfaces: auth gates, secrets hygiene, injection and unsafe sinks, unsafe deserialization/crypto/transport, and LLM tool-use trust boundaries. Reports only, never fixes. Use when the user asks for a security review or to check vulnerabilities, secrets, or auth.

★ 2