qa-verdict-gate
Kill-gate: QA PASS/FAIL/INCONCLUSIVE + qa-report.md. Triggers: QA verdict, QA done/complete, pre-merge QA, false-FAIL.
- License 许可
- Apache-2.0
- 所属仓库
- EtanHey/golems
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- qa-verdict-gate
- description
- Kill-gate: QA PASS/FAIL/INCONCLUSIVE + qa-report.md. Triggers: QA verdict, QA done/complete, pre-merge QA, false-FAIL.
- disable-model-invocation
- true
02 资源文件 41
- 参考文档evals/fixtures/green/01-fail-observed-error-with-report.json1.9 KB
- 参考文档evals/fixtures/green/02-inconclusive-unreachable.json1.5 KB
- 参考文档evals/fixtures/green/03-pass-full-report.json1.8 KB
- 参考文档evals/fixtures/green/04-in-progress-na.json460 B
- 参考文档evals/fixtures/green/05-verdict-pending-na.json531 B
- 参考文档evals/fixtures/green/06-decision-claim-with-visual-evidence.json1.6 KB
- 参考文档evals/fixtures/green/07-generic-decision-word-not-etan-decision.json1.3 KB
- 参考文档evals/fixtures/green/08-quoted-verdict-word-not-a-qa-claim.json810 B
- 参考文档evals/fixtures/green/09-prose-blocked-and-pass-no-qa-context.json1.1 KB
- 参考文档evals/fixtures/green/10-video-inspected-frames.json1.8 KB
- 参考文档evals/fixtures/green/11-video-ledger-only.json1.6 KB
- 参考文档evals/fixtures/green/12-video-inspected-digest.json1.9 KB
- 参考文档evals/fixtures/red/01-fail-page-never-loaded.json1.2 KB
- 参考文档evals/fixtures/red/02-complete-no-report.json944 B
- 参考文档evals/fixtures/red/03-fail-no-observation.json1.2 KB
- 参考文档evals/fixtures/red/04-prose-bypass-observed-error.json1.0 KB
- 参考文档evals/fixtures/red/05-passive-read-supplies-error.json1.2 KB
- 参考文档evals/fixtures/red/06-pass-unreachable.json1.4 KB
- 参考文档evals/fixtures/red/07-report-stub-one-item.json1.2 KB
- 参考文档evals/fixtures/red/08-fail-prose-unreached.json1004 B
- 参考文档evals/fixtures/red/09-namespaced-write-fail-no-observation.json1018 B
- 参考文档evals/fixtures/red/10-fail-navigate-only.json1.2 KB
- 参考文档evals/fixtures/red/11-decision-claim-no-visual-reference.json1.5 KB
- 参考文档evals/fixtures/red/12-user-facing-ready-merge-only.json1.4 KB
- 参考文档evals/fixtures/red/13-video-no-ledger.json1.5 KB
- 参考文档evals/fixtures/red/14-video-ledger-in-prose.json1.8 KB
- 参考文档evals/fixtures/red/15-video-different-finding.json1.8 KB
- 参考文档evals/fixtures/red/16-video-prior-turn.json1.9 KB
- 参考文档evals/fixtures/red/17-video-different-run.json1.8 KB
- 参考文档evals/fixtures/red/18-video-malformed-ledger.json1.8 KB
- 参考文档evals/fixtures/red/19-video-unobserved-second-finding.json1.9 KB
- 参考文档evals/fixtures/red/20-video-traversal-frame.json1.8 KB
- 参考文档evals/fixtures/red/21-video-confirmation-only.json1.7 KB
- 参考文档evals/fixtures/red/22-video-inspected-malformed-digest.json1.9 KB
- 脚本evals/qa-verdict-gate.test.mjs5.1 KB
- 脚本evals/run_suite.py6.7 KB
- 参考文档install-snippet.json304 B
- 脚本lib/transcript.mjs6.7 KB
- 脚本scripts/qa-verdict-gate-cli.mjs1.6 KB
- 脚本scripts/qa-verdict-gate-hook.mjs1.6 KB
- 脚本src/qa-verdict-gate.mjs24.4 KB
03 SKILL.md 正文
Skill: QA Verdict-Integrity Kill-Gate (gen-18 Track 2 #6)
Fleet law: canon #4 owns user-visible completion; canon #3 owns evidence. This skill enforces QA verdict integrity, report artifacts, and rendered-state evidence.
Scope
FAIL is only for confirmed-observed failures; INCONCLUSIVE is for when the path couldn't be reached. A qa-report.md artifact must exist.
What It Is
A deterministic detector over a QA transcript/verdict: a QA result is a verdict-integrity violation
unless the agent's stated verdict matches the OBSERVED evidence in the same turn AND a terminal
qa-report.md artifact exists. This is the MECHANICAL gate that /qa-video and /never-fabricate
describe in prose. The pinned RED/GREEN transcript fixtures ARE the replayable gate (R-003/R-014 pattern,
T6 deterministic smoke-spec shape). Sibling of /false-green-gate — same lib/transcript.mjs, same
buildEvidence discipline (evidence comes from tool calls + tool_result OUTPUTS, never assistant prose).
The Rule — tri-state, evidence-anchored
| Detector code | Fires when | The honest verdict instead |
|---|---|---|
QA_FAIL_WITHOUT_OBSERVATION |
a FAIL with no same-turn observed-failure evidence (screenshot, click that reached the surface, observed error, or matching qa-video inspected-ledger rows in tool_result outputs) |
INCONCLUSIVE |
QA_UNREACHED_NOT_INCONCLUSIVE |
the transcript shows the path couldn't be reached ("couldn't load", "element not found", "blocked at step 0", net::ERR_*), but the verdict is FAIL or PASS |
INCONCLUSIVE |
QA_NO_REPORT_ARTIFACT |
a QA "done/complete" claim with no qa-report.md written this turn carrying the checklist items (≥2) |
write the report first |
QA_DECISION_CLAIM_NO_VISUAL_EVIDENCE |
a terminal QA/completion answer that feeds an Etan merge/ship/QA decision has no screenshot/dashboard/rendered-state reference embedded in that same answer | embed the visual evidence reference where Etan answers |
QA_USER_VISIBLE_WITHOUT_RENDERED_EVIDENCE |
a PASS/ready/done verdict about a user-facing surface is backed only by merge/plumbing state, with no same-turn screenshot/click/rendered page/HTTP 2xx probe | probe the rendered user-visible surface first |
PASS = the stated verdict matches the observed evidence and the qa-report.md artifact exists.
"Same turn" = the events since the last human message — the probe tool calls and their results the
agent ran before its verdict. Negated/in-progress statements ("not done yet") are not verdicts. An explicit
Verdict: INCONCLUSIVE line is authoritative over loose pass/fail words elsewhere in the prose.
Recorded-frame observation
For qa-video, same-turn tool outputs may supply headerless inspected-ledger rows:
pts<TAB>absolute frame path<TAB>reader: why<TAB>finding-id[<TAB>sha256].
Legacy four-column rows are accepted. Five-column rows require a 64-hex-digit
SHA-256 field; other row shapes are rejected. Read the run's
inspected.tsv (or logs/inspected*.tsv) and include its rows in the tool output.
PTS must be finite and nonnegative; the canonical absolute .png/.jpg/.jpeg
path must stay under the run directory of a same-turn Write/Edit/MultiEdit to
/…/qa-video/<run>/qa-report.md (namespaced tool names are supported).
There must be exactly one such run root. Every named FAIL in assistant text or
the report needs a matching row: F1 FAIL or ### F1: title — **FAIL**, with
case-sensitive alphanumeric/hyphen IDs containing a digit (e.g. S1-F6r).
Rows for another finding/run, earlier-turn rows, assistant prose, and CONFIRMED
records or verification summaries alone do not count. This checks inspection
receipts, not image contents, before/after sufficiency, or verifier correctness.
Only the FAIL observation check accepts these receipts; the other four checks
remain unchanged, including the requirement to write the report this turn.
Why FAIL is reserved
A false-FAIL (codex 019ee5ea#1) and a truncated QA with no artifact (019ee493#2/#4) are the two
specimens this closes. A FAIL emitted when the regression path was never reached is worse than no QA —
it sends a worker chasing a phantom bug. FAIL must come from a confirmed observation; everything the run
could not actually exercise is INCONCLUSIVE, surfaced honestly.
How /pr-loop + /qa-video Consume It
Before a QA seat emits a verdict or a "QA complete" message, run the gate on the turn — /qa-verdict-gate
(bun skills/golem-powers/qa-verdict-gate/scripts/qa-verdict-gate-cli.mjs <transcript|->, exit 3 = FLAG).
A FLAG means the verdict is not yet earned: reach the surface and observe, or downgrade FAIL→INCONCLUSIVE,
write the missing qa-report.md, embed the visual reference in decision-feeding answers, or probe the
rendered user-visible surface. Compose with /false-green-gate (live-outcome probe for completion claims)
and /never-fabricate (read the output before reporting).
Run It
bun test skills/golem-powers/qa-verdict-gate/evals/qa-verdict-gate.test.mjs # replay (CI-safe)
python3 skills/golem-powers/qa-verdict-gate/evals/run_suite.py # hook schema + fail-open
bun skills/golem-powers/qa-verdict-gate/scripts/qa-verdict-gate-cli.mjs <transcript.jsonl|->
Programmatic: import { detectQaVerdict } from "./src/qa-verdict-gate.mjs" → { verdict, claim, qaVerdict, violations }.
Stated Limits (honesty rule)
- Evidence is marker-anchored over the same-turn blob; a probe described but not actually run can read as present. The fixtures pin the known specimens; new evasion shapes are added as RED fixtures (R-003 model).
- "Same turn" is bounded by the last human message; a probe run two turns earlier does not count (by design).
- Observed-error / screenshot tokens in a
tool_resultonly count when a Bash command or a browser tool actually drove a surface this turn — a passiveReadof an old log cannot supply them (same-turn-scope). The recorded-frame route instead accepts a same-turn ledger Read under the contract above. - Decision-feeding evidence is checked on the terminal answer text: a screenshot mentioned only earlier in the turn does not help Etan answer the decision prompt.
Provenance
RED specimens: codex/019ee5ea#1 (false-FAIL, path unreachable), codex/019ee493#2/#4 (truncated QA,
no artifact), a FAIL with zero observed-failure evidence. Evasion REDs: prose-bypass (narrative "observed
error"), passive-read-supplies-error (same-turn-scope), false-PASS over an unreachable surface,
report-stub (artifact with no checklist), prose-unreached-still-FAIL (negation), namespaced-MCP-Write
(tool-name normalization), navigate-only-clears-FAIL (cursor-bugbot L195 — bare browser_navigate does
not exercise the failing flow), decision-feeding bare verdict with no visual evidence reference,
user-facing dashboard ready claim backed only by merge state. GREEN references: FAIL with an observed-error
screenshot + report, INCONCLUSIVE correctly used when unreachable, PASS with a full report, in-progress
(N/A), verdict-pending (cursor-bugbot L260 — "step 1 done, verdict still pending" is not a settled verdict),
decision-feeding claim with the screenshot/dashboard reference embedded in the same terminal answer.
04 同仓库更多 Skills
yash
Push dependency bugs upstream. Triggers: dependency bug, workaround, monkey-patch, fork/fix. NOT app bugs.
whats-new
Diff Claude/Codex/Cursor/Wispr/agy changelogs vs local setup. Triggers: whats new, changelog, version bump.
weave
Orchestrator convergence: mine JSONLs, cite findings, route actions. Triggers: weave, run weave.
unslop
Cut AI tells while keeping every receipt, number, path, and hedge intact. Triggers: unslop, cut the slop, wall of text, too verbose, rambly, make this readable. NOT for delivery cadence (see i-have-adhd), channel choice, or artifact length governance.
tmp-block
Guard denying durable writes to /tmp, /private/tmp, /var/folders, $TMPDIR; enforces <repo>/.worktrees/<name>. Triggers: tmp guard, TMP-BLOCK deny, WORKTREE-CONVENTION deny, bypass ledger. NOT for storage policy.
t3-visuals
Show/visualize/chart/diagram/compare/status board in T3 with compact interactive HTML. Not standalone dashboards.