[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6DDSJ2MKBF8H7K221C7CF5H":3,"skill-same:01M6DDSJ2MKBF8H7K221C7CF5H":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":14,"repositoryHtmlUrl":51,"repositoryHomepageUrl":6,"ownerHtmlUrl":52,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":53,"install":54,"resources":67},"01M6DDSJ2MKBF8H7K221C7CF5H","agent-as-judge",null,"Advisory, evidence-citing Agent-as-a-Judge. Use for a second opinion on whether a completion meets its Definition of Done AFTER the deterministic verification cluster (typed receipts, fail-first, execution-consensus, golden-journey oracle) ran - decomposes the DoD into falsifiable YES\u002FNO items, cites evidence per item, scores on ICD-203. 100% ADVISORY\u002FSHADOW, never blocks. NOT a done-gate.","Prekzursil\u002Fagent-skills-toolchain","Prekzursil","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F54636077?v=4","01M6DACZ1PCQHD99SCR1YXG28B","Andrei Mihai Visalon","01M6DACZ670WZF535V3WYYSS4Z",0,"MIT","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Fskills\u002Fagent-as-judge\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"agent as judge","as","never","advisory","done","deterministic","journey","gate","Shadow","evidence","oracle","use","consensus","receipts","first","execution","verification","second opinion","definition of done","2026-11-10 10:12:30",false,"---\nname: agent-as-judge\ndescription: Advisory, evidence-citing Agent-as-a-Judge. Use for a second opinion on whether a completion meets its Definition of Done AFTER the deterministic verification cluster (typed receipts, fail-first, execution-consensus, golden-journey oracle) ran - decomposes the DoD into falsifiable YES\u002FNO items, cites evidence per item, scores on ICD-203. 100% ADVISORY\u002FSHADOW, never blocks. NOT a done-gate.\n---\n# Agent-as-a-Judge (advisory, evidence-citing)\n\nA tool-using LLM judge that annotates the **subjective residue** of a Definition of Done — the\nitems the deterministic gates cannot decide (\"is the artifact actually correct?\", \"does this\nmatch intent?\"). It is **advisory this wave**: its opinion never gates, and it ships in SHADOW\nmode. The subagent contract lives in `agents\u002Fagent-as-judge.md`; this skill is the operator guide\nplus the reference implementation the judge runs.\n\n**Evidence:** MASTER-DECISION-QUEUE A5 (`gem-hunt-2026-07-12`); COUNCIL-VERDICT C3\n(*\"opinion never gates, only a cited executed receipt can block — deferred indefinitely\"*).\nResearch: Agent-as-a-Judge (arXiv 2410.10934); ICD-203 Analytic Standards; EvalGen (2404.12272);\nJudgeBench (LLM judges ~chance) — the empirical basis for the advisory-until-calibrated stance.\n\n## Where it sits in the pipeline (read this first)\n\n```\n   deterministic cluster (carries ALL blocking weight, no calibration needed)\n   ┌─────────────┬──────────────┬────────────────────┬─────────────────────┐\n   │ A1 typed    │ A3 fail-first│ A4 execution-      │ C4 golden-journey   │\n   │ receipts    │ red-proof    │ consensus          │ oracle (keystone)   │\n   └─────────────┴──────────────┴────────────────────┴─────────────────────┘\n                                   │  green?\n                                   ▼\n              ┌─────────────────────────────────────────────┐\n              │  Agent-as-a-Judge  (THIS)  — runs AFTER      │\n              │  ADVISORY · SHADOW · annotates SUBJECTIVE DoD │\n              │  opinion NEVER blocks\u002Fgrants · cites+executes │\n              └─────────────────────────────────────────────┘\n```\n\nIf the deterministic cluster has not run green, the judge says so and **lowers its confidence** —\nit does not substitute its opinion for the missing deterministic signal.\n\n## When to use\n\n- After a work-unit's deterministic gates pass and you want an evidence-cited second read on the\n  *subjective* DoD items before (not instead of) an owner sign-off.\n- To accrue a **calibration record** — every advisory verdict is logged so `passk-stats` (F5) can\n  later measure the judge's agreement (kappa) with the owner's own corrections.\n- To decide whether a completion is contested\u002Firreversible enough to warrant a **cross-vendor\n  Codex second judge**.\n\n## When NOT to use\n\n- To **block** or **clear** a merge\u002Fcompletion — it structurally cannot (COUNCIL C3). Use the\n  deterministic cluster for the done-signal.\n- As the primary verifier — it is one correlated-failure-prone vote, never terminal authority.\n- On purely deterministic criteria the gates already cover — the judge only owns the residue.\n\n## The contract in one screen\n\n| Rule | Consequence |\n|------|-------------|\n| Bare opinion never gates\u002Fgrants \"done\" | Output is 100% advisory this wave |\n| Only a cited, re-run EXECUTED receipt could ever block | That veto path is **deferred indefinitely** |\n| Runs AFTER the deterministic cluster | It owns only the subjective residue |\n| Veto layer, never grant layer | May add scrutiny; may never clear |\n| Same-vendor = correlated failure | Prefer a cross-vendor Codex second judge |\n| Calibrate kappa >= 0.6 vs claude-reflect corpus, per-category, lower CI bound | Even then: license to SPEAK, not BLOCK |\n| Honest prior kappa \u003C 0.6 | Stays advisory; program loses nothing |\n| Auto-demote on kappa decay; never self-promote | A good judge can rot |\n| Read-only on source; writes only the shadow sink | Never edits project code |\n\n## Protocol (four steps)\n\n1. **Extract a YES\u002FNO DoD checklist** from `ACCEPTANCE.md` \u002F issue \u002F plan \u002F `active-plan.md`;\n   split compound criteria; tag each `deterministic` (a gate covered it) or `subjective` (judge\n   owns it). Opine only on `subjective` items.\n2. **Cite evidence per item** — `ledger#seq\u003CN>` (a real event in\n   `~\u002F.claude\u002Fevent-ledger\u002F\u003Cslug>\u002Fevents.jsonl`) OR `sha256:\u003Chex12>` of the exact artifact bytes \u002F\n   command output (chain-of-custody). Un-citable item -> `UNBACKED` -> advisory-forever.\n3. **Execute-to-confirm** every would-be-blocking finding: re-run the cited receipt; record\n   `reproduced` \u002F `not-reproduced` (-> downgrade to note) \u002F `unrunnable` (-> UNBACKED).\n4. **Score ICD-203's nine standards** (sourcing, uncertainty, fact-vs-assumption, alternatives,\n   relevance, logic, consistency, likelihood, presentation) PASS\u002FPARTIAL\u002FFAIL each.\n\nThen emit the terminal-state verdict (schema in `agents\u002Fagent-as-judge.md`) and append one shadow\nline. **Zero findings block — that is by design.**\n\n## Calibration harness (passk-stats \u002F F5)\n\n- **Label source:** owner's claude-reflect corrections `~\u002F.claude\u002Flearnings-queue.json` +\n  `--supersedes` chains in the ledger. This is a *caught-error* corpus — blind to *escaped*\n  defects — so it can license the judge to SPEAK, never to BLOCK on opinion (C3 hard constraint).\n- **Statistic:** Cohen's kappa (>= 0.6 substantial) + anytime-valid e-values \u002F paired CI via\n  `passk-stats`, over >= 30-50 labeled items, gated on the LOWER CI bound, per-DoD-category.\n- **F5 not built yet?** `probe_passk_stats()` returns absent -> `CALIBRATION: UNAVAILABLE` ->\n  advisory. Never fabricate kappa; a missing calibrator can never flip the judge toward gating.\n\n## Shadow-mode logging (COUNCIL C5)\n\nOne JSON line per run to a **separate, bounded, auto-rotating** sink — never the main ledger:\n\n- Default sink: `~\u002F.claude\u002Fshadow-sinks\u002Fagent-as-judge.jsonl` (override `$AGENT_AS_JUDGE_SHADOW_SINK`).\n- Rotates at ~2 MB to `\u003Csink>.1`. Records verdict + would-have-fired findings + citations +\n  calibration status so F1 (gem-fitness scorer) can measure precision against external truth.\n- SHADOW = observe\u002Flog only. It NEVER blocks and NEVER nags. There is no ENFORCE promotion this\n  wave; promotion is an owner decision gated on external calibration.\n\n## Reference implementation\n\nStdlib-only, fail-open, Windows\u002FPOSIX-safe. The judge runs this via Bash — do not re-implement it\nfrom memory. Save as needed or run inline. Modes: `--selftest` (runs against the live ledger +\nlabel corpus and prints a demo verdict) and library use (`import` the functions).\n\n```python\n#!\u002Fusr\u002Fbin\u002Fenv python\n\"\"\"agent-as-judge helper — evidence citation, evidence hashing, label-corpus read,\npassk-stats (F5) probe, and bounded rotating shadow-log. Stdlib only; fail-open.\n\nCONTRACT: advisory-only. Nothing here blocks, gates, or grants 'done'. It records and cites.\n\"\"\"\nfrom __future__ import annotations\nimport hashlib, json, os, sys, time\nfrom pathlib import Path\n\nHOME = Path(os.path.expanduser(\"~\"))\nLEDGER_MOD_DIR = HOME \u002F \".claude\" \u002F \"hooks\" \u002F \"event-ledger\"\nLABEL_CORPUS = HOME \u002F \".claude\" \u002F \"learnings-queue.json\"          # claude-reflect corrections\nDEFAULT_SINK = HOME \u002F \".claude\" \u002F \"shadow-sinks\" \u002F \"agent-as-judge.jsonl\"\nSINK = Path(os.environ.get(\"AGENT_AS_JUDGE_SHADOW_SINK\", str(DEFAULT_SINK)))\nSINK_ROTATE_BYTES = 2 * 1024 * 1024\nKAPPA_BAR = 0.6                    # Landis-Koch \"substantial\"\nMIN_LABELED = 30                   # C3: >= 30-50 labeled items before kappa is even meaningful\n\n\ndef _load_ledger():\n    \"\"\"Best-effort import of the real event-ledger module. Fail-open -> None.\"\"\"\n    try:\n        if str(LEDGER_MOD_DIR) not in sys.path:\n            sys.path.insert(0, str(LEDGER_MOD_DIR))\n        import ledger  # noqa: PLC0415\n        return ledger\n    except Exception:\n        return None\n\n\ndef cite_ledger(project_path: str, n: int = 40) -> list[dict]:\n    \"\"\"Return recent ledger events (oldest->newest) for citation as ledger#seq\u003CN>.\"\"\"\n    led = _load_ledger()\n    if led is None:\n        return []\n    try:\n        return led.replay(project_path or os.getcwd(), n=n)\n    except Exception:\n        return []\n\n\ndef ledger_citation(ev: dict) -> str:\n    \"\"\"Format a real event as a chain-of-custody citation.\"\"\"\n    return f\"ledger#seq{ev.get('seq')}({ev.get('kind')}@{ev.get('ts')})\"\n\n\ndef hash_evidence(source, is_path: bool | None = None) -> str:\n    \"\"\"sha256:\u003Chex12> of artifact BYTES (path) or command-output TEXT. Pins what you saw.\"\"\"\n    try:\n        if is_path is None:\n            is_path = isinstance(source, (str, Path)) and Path(str(source)).exists()\n        if is_path:\n            data = Path(str(source)).read_bytes()\n        else:\n            data = str(source).encode(\"utf-8\", \"replace\")\n        return \"sha256:\" + hashlib.sha256(data).hexdigest()[:12]\n    except Exception:\n        return \"sha256:UNREADABLE\"\n\n\ndef read_label_corpus() -> dict:\n    \"\"\"Read the owner's claude-reflect corrections queue = kappa ground-truth labels.\n    Caught-error corpus (blind to escaped defects) -> licenses SPEAK, never BLOCK (C3).\"\"\"\n    if not LABEL_CORPUS.exists():\n        return {\"present\": False, \"n\": 0, \"reason\": \"learnings-queue.json absent (no corrections captured yet)\"}\n    try:\n        items = json.loads(LABEL_CORPUS.read_text(encoding=\"utf-8\"))\n        corrections = [it for it in items if isinstance(it, dict)\n                       and (it.get(\"sentiment\") == \"correction\" or it.get(\"type\") in (\"auto\", \"explicit\", \"guardrail\"))]\n        return {\"present\": True, \"n\": len(corrections), \"total\": len(items)}\n    except Exception as e:\n        return {\"present\": False, \"n\": 0, \"reason\": f\"unreadable: {e!r}\"}\n\n\ndef probe_passk_stats() -> dict:\n    \"\"\"Detect the F5 pass^k \u002F e-value \u002F kappa calibrator. Absent -> advisory, never fabricate.\"\"\"\n    # F5 may ship as a skill dir, a module, or an artifact. Probe the known shapes.\n    candidates = [\n        HOME \u002F \".claude\" \u002F \"skills\" \u002F \"passk-stats\" \u002F \"SKILL.md\",\n        HOME \u002F \".claude\" \u002F \"skills\" \u002F \"pass-k-stats\" \u002F \"SKILL.md\",\n        HOME \u002F \".claude\" \u002F \"hooks\" \u002F \"passk_stats.py\",\n    ]\n    for c in candidates:\n        if c.exists():\n            return {\"present\": True, \"path\": str(c)}\n    try:\n        import passk_stats  # noqa: F401,PLC0415\n        return {\"present\": True, \"path\": \"module:passk_stats\"}\n    except Exception:\n        return {\"present\": False}\n\n\ndef calibration_status(labels: dict, passk: dict) -> str:\n    \"\"\"The honest, non-fabricating calibration verdict. Never returns a made-up kappa.\"\"\"\n    if not passk.get(\"present\"):\n        return \"CALIBRATION: UNAVAILABLE (passk-stats\u002FF5 not present) -> advisory\"\n    if not labels.get(\"present\") or labels.get(\"n\", 0) \u003C MIN_LABELED:\n        return (f\"CALIBRATION: INSUFFICIENT LABELS (n={labels.get('n', 0)} \u003C {MIN_LABELED}) \"\n                f\"-> advisory (honest prior kappa \u003C {KAPPA_BAR})\")\n    # F5 present AND enough labels: hand off to passk-stats to compute per-category kappa + e-value.\n    # This helper does NOT compute kappa itself (that is F5's job); it reports the handoff.\n    return (f\"CALIBRATION: DELEGATE to passk-stats over {labels['n']} labels \"\n            f\"(per-category kappa, lower-CI-bound gate, bar={KAPPA_BAR}); stays advisory until owner promotes\")\n\n\ndef shadow_log(record: dict) -> str:\n    \"\"\"Append ONE JSON line to the bounded, rotating shadow sink. Never the main ledger.\"\"\"\n    try:\n        SINK.parent.mkdir(parents=True, exist_ok=True)\n        if SINK.exists() and SINK.stat().st_size > SINK_ROTATE_BYTES:\n            rotated = SINK.with_suffix(SINK.suffix + \".1\")\n            try:\n                if rotated.exists():\n                    rotated.unlink()\n                SINK.rename(rotated)\n            except Exception:\n                pass\n        record = {\"ts\": time.strftime(\"%Y-%m-%dT%H:%M:%S\"), \"mode\": \"SHADOW\u002FADVISE\",\n                  \"blocks\": False, **record}\n        with SINK.open(\"a\", encoding=\"utf-8\") as fh:\n            fh.write(json.dumps(record, ensure_ascii=False) + \"\\n\")\n        return str(SINK)\n    except Exception as e:\n        return f\"SHADOW-LOG-FAILED: {e!r}\"\n\n\ndef selftest() -> int:\n    \"\"\"Run against the LIVE ledger + label corpus; emit a demo advisory verdict. Blocks nothing.\"\"\"\n    project = os.environ.get(\"CLAUDE_CWD\", os.getcwd())\n    events = cite_ledger(project, n=10)\n    labels = read_label_corpus()\n    passk = probe_passk_stats()\n    cal = calibration_status(labels, passk)\n\n    # Demo DoD item cited two legal ways (ledger event if any exist, else an evidence hash).\n    if events:\n        item_cite = ledger_citation(events[-1])\n        cite_kind = \"ledger-event\"\n    else:\n        item_cite = hash_evidence(\"DEMO: no ledger events yet for this project\", is_path=False)\n        cite_kind = \"evidence-hash (no ledger events -> hashed the observation)\"\n\n    verdict = {\n        \"verdict\": \"ADVISORY (does not gate)\",\n        \"ran_after_deterministic_cluster\": \"unknown (selftest)\",\n        \"checklist_demo\": [{\n            \"item\": \"DoD item extracted (demo)\", \"type\": \"subjective\", \"opinion\": \"YES\",\n            \"citation\": item_cite, \"citation_kind\": cite_kind, \"receipt\": \"unrunnable (demo)\",\n        }],\n        \"icd203\": \"9-standard rubric applied per item (see agents\u002Fagent-as-judge.md)\",\n        \"calibration\": cal,\n        \"ledger_events_available_for_citation\": len(events),\n        \"label_corpus\": labels,\n        \"passk_stats\": passk,\n        \"blocking_findings\": 0,\n    }\n    sink = shadow_log({\"event\": \"selftest\", \"verdict\": verdict})\n    print(\"=== Agent-as-a-Judge selftest (ADVISORY — blocks nothing) ===\")\n    print(json.dumps(verdict, ensure_ascii=False, indent=2))\n    print(f\"SHADOW-LOG -> {sink}\")\n    print(\"SUCCESS:agent-as-judge selftest — advisory verdict emitted, 0 blocking findings\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(selftest() if \"--selftest\" in sys.argv else selftest())\n```\n\n## Honest limits\n\n- **Not a done-signal.** By council law it cannot block or clear; the deterministic cluster owns\n  the done-signal. If someone wires this into a blocking path, that violates C3.\n- **Correlated failure.** A same-vendor judge shares the fixer's blind spots; a cross-vendor Codex\n  second judge is the independent check for contested\u002Firreversible completions.\n- **Calibration corpus is caught-error-only.** kappa vs claude-reflect corrections optimizes for\n  re-catching *known* error classes; it is blind to escaped defects, so it can never license\n  opinion-gating (only a fresh prospective held-out set the judge never saw could — deferred).\n- **F5 dependency.** kappa\u002Fe-value calibration requires `passk-stats`; until it exists the judge is\n  advisory with `CALIBRATION: UNAVAILABLE`. That is the correct, safe default — not a bug.\n","# Agent-as-a-Judge (advisory, evidence-citing)\n\nA tool-using LLM judge that annotates the **subjective residue** of a Definition of Done — the\nitems the deterministic gates cannot decide (\"is the artifact actually correct?\", \"does this\nmatch intent?\"). It is **advisory this wave**: its opinion never gates, and it ships in SHADOW\nmode. The subagent contract lives in `agents\u002Fagent-as-judge.md`; this skill is the operator guide\nplus the reference implementation the judge runs.\n\n**Evidence:** MASTER-DECISION-QUEUE A5 (`gem-hunt-2026-07-12`); COUNCIL-VERDICT C3\n(*\"opinion never gates, only a cited executed receipt can block — deferred indefinitely\"*).\nResearch: Agent-as-a-Judge (arXiv 2410.10934); ICD-203 Analytic Standards; EvalGen (2404.12272);\nJudgeBench (LLM judges ~chance) — the empirical basis for the advisory-until-calibrated stance.\n\n## Where it sits in the pipeline (read this first)\n\n```\n   deterministic cluster (carries ALL blocking weight, no calibration needed)\n   ┌─────────────┬──────────────┬────────────────────┬─────────────────────┐\n   │ A1 typed    │ A3 fail-first│ A4 execution-      │ C4 golden-journey   │\n   │ receipts    │ red-proof    │ consensus          │ oracle (keystone)   │\n   └─────────────┴──────────────┴────────────────────┴─────────────────────┘\n                                   │  green?\n                                   ▼\n              ┌─────────────────────────────────────────────┐\n              │  Agent-as-a-Judge  (THIS)  — runs AFTER      │\n              │  ADVISORY · SHADOW · annotates SUBJECTIVE DoD │\n              │  opinion NEVER blocks\u002Fgrants · cites+executes │\n              └─────────────────────────────────────────────┘\n```\n\nIf the deterministic cluster has not run green, the judge says so and **lowers its confidence** —\nit does not substitute its opinion for the missing deterministic signal.\n\n## When to use\n\n- After a work-unit's deterministic gates pass and you want an evidence-cited second read on the\n  *subjective* DoD items before (not instead of) an owner sign-off.\n- To accrue a **calibration record** — every advisory verdict is logged so `passk-stats` (F5) can\n  later measure the judge's agreement (kappa) with the owner's own corrections.\n- To decide whether a completion is contested\u002Firreversible enough to warrant a **cross-vendor\n  Codex second judge**.\n\n## When NOT to use\n\n- To **block** or **clear** a merge\u002Fcompletion — it structurally cannot (COUNCIL C3). Use the\n  deterministic cluster for the done-signal.\n- As the primary verifier — it is one correlated-failure-prone vote, never terminal authority.\n- On purely deterministic criteria the gates already cover — the judge only owns the residue.\n\n## The contract in one screen\n\n| Rule | Consequence |\n|------|-------------|\n| Bare opinion never gates\u002Fgrants \"done\" | Output is 100% advisory this wave |\n| Only a cited, re-run EXECUTED receipt could ever block | That veto path is **deferred indefinitely** |\n| Runs AFTER the deterministic cluster | It owns only the subjective residue |\n| Veto layer, never grant layer | May add scrutiny; may never clear |\n| Same-vendor = correlated failure | Prefer a cross-vendor Codex second judge |\n| Calibrate kappa >= 0.6 vs claude-reflect corpus, per-category, lower CI bound | Even then: license to SPEAK, not BLOCK |\n| Honest prior kappa \u003C 0.6 | Stays advisory; program loses nothing |\n| Auto-demote on kappa decay; never self-promote | A good judge can rot |\n| Read-only on source; writes only the shadow sink | Never edits project code |\n\n## Protocol (four steps)\n\n1. **Extract a YES\u002FNO DoD checklist** from `ACCEPTANCE.md` \u002F issue \u002F plan \u002F `active-plan.md`;\n   split compound criteria; tag each `deterministic` (a gate covered it) or `subjective` (judge\n   owns it). Opine only on `subjective` items.\n2. **Cite evidence per item** — `ledger#seq\u003CN>` (a real event in\n   `~\u002F.claude\u002Fevent-ledger\u002F\u003Cslug>\u002Fevents.jsonl`) OR `sha256:\u003Chex12>` of the exact artifact bytes \u002F\n   command output (chain-of-custody). Un-citable item -> `UNBACKED` -> advisory-forever.\n3. **Execute-to-confirm** every would-be-blocking finding: re-run the cited receipt; record\n   `reproduced` \u002F `not-reproduced` (-> downgrade to note) \u002F `unrunnable` (-> UNBACKED).\n4. **Score ICD-203's nine standards** (sourcing, uncertainty, fact-vs-assumption, alternatives,\n   relevance, logic, consistency, likelihood, presentation) PASS\u002FPARTIAL\u002FFAIL each.\n\nThen emit the terminal-state verdict (schema in `agents\u002Fagent-as-judge.md`) and append one shadow\nline. **Zero findings block — that is by design.**\n\n## Calibration harness (passk-stats \u002F F5)\n\n- **Label source:** owner's claude-reflect corrections `~\u002F.claude\u002Flearnings-queue.json` +\n  `--supersedes` chains in the ledger. This is a *caught-error* corpus — blind to *escaped*\n  defects — so it can license the judge to SPEAK, never to BLOCK on opinion (C3 hard constraint).\n- **Statistic:** Cohen's kappa (>= 0.6 substantial) + anytime-valid e-values \u002F paired CI via\n  `passk-stats`, over >= 30-50 labeled items, gated on the LOWER CI bound, per-DoD-category.\n- **F5 not built yet?** `probe_passk_stats()` returns absent -> `CALIBRATION: UNAVAILABLE` ->\n  advisory. Never fabricate kappa; a missing calibrator can never flip the judge toward gating.\n\n## Shadow-mode logging (COUNCIL C5)\n\nOne JSON line per run to a **separate, bounded, auto-rotating** sink — never the main ledger:\n\n- Default sink: `~\u002F.claude\u002Fshadow-sinks\u002Fagent-as-judge.jsonl` (override `$AGENT_AS_JUDGE_SHADOW_SINK`).\n- Rotates at ~2 MB to `\u003Csink>.1`. Records verdict + would-have-fired findings + citations +\n  calibration status so F1 (gem-fitness scorer) can measure precision against external truth.\n- SHADOW = observe\u002Flog only. It NEVER blocks and NEVER nags. There is no ENFORCE promotion this\n  wave; promotion is an owner decision gated on external calibration.\n\n## Reference implementation\n\nStdlib-only, fail-open, Windows\u002FPOSIX-safe. The judge runs this via Bash — do not re-implement it\nfrom memory. Save as needed or run inline. Modes: `--selftest` (runs against the live ledger +\nlabel corpus and prints a demo verdict) and library use (`import` the functions).\n\n```python\n#!\u002Fusr\u002Fbin\u002Fenv python\n\"\"\"agent-as-judge helper — evidence citation, evidence hashing, label-corpus read,\npassk-stats (F5) probe, and bounded rotating shadow-log. Stdlib only; fail-open.\n\nCONTRACT: advisory-only. Nothing here blocks, gates, or grants 'done'. It records and cites.\n\"\"\"\nfrom __future__ import annotations\nimport hashlib, json, os, sys, time\nfrom pathlib import Path\n\nHOME = Path(os.path.expanduser(\"~\"))\nLEDGER_MOD_DIR = HOME \u002F \".claude\" \u002F \"hooks\" \u002F \"event-ledger\"\nLABEL_CORPUS = HOME \u002F \".claude\" \u002F \"learnings-queue.json\"          # claude-reflect corrections\nDEFAULT_SINK = HOME \u002F \".claude\" \u002F \"shadow-sinks\" \u002F \"agent-as-judge.jsonl\"\nSINK = Path(os.environ.get(\"AGENT_AS_JUDGE_SHADOW_SINK\", str(DEFAULT_SINK)))\nSINK_ROTATE_BYTES = 2 * 1024 * 1024\nKAPPA_BAR = 0.6                    # Landis-Koch \"substantial\"\nMIN_LABELED = 30                   # C3: >= 30-50 labeled items before kappa is even meaningful\n\n\ndef _load_ledger():\n    \"\"\"Best-effort import of the real event-ledger module. Fail-open -> None.\"\"\"\n    try:\n        if str(LEDGER_MOD_DIR) not in sys.path:\n            sys.path.insert(0, str(LEDGER_MOD_DIR))\n        import ledger  # noqa: PLC0415\n        return ledger\n    except Exception:\n        return None\n\n\ndef cite_ledger(project_path: str, n: int = 40) -> list[dict]:\n    \"\"\"Return recent ledger events (oldest->newest) for citation as ledger#seq\u003CN>.\"\"\"\n    led = _load_ledger()\n    if led is None:\n        return []\n    try:\n        return led.replay(project_path or os.getcwd(), n=n)\n    except Exception:\n        return []\n\n\ndef ledger_citation(ev: dict) -> str:\n    \"\"\"Format a real event as a chain-of-custody citation.\"\"\"\n    return f\"ledger#seq{ev.get('seq')}({ev.get('kind')}@{ev.get('ts')})\"\n\n\ndef hash_evidence(source, is_path: bool | None = None) -> str:\n    \"\"\"sha256:\u003Chex12> of artifact BYTES (path) or command-output TEXT. Pins what you saw.\"\"\"\n    try:\n        if is_path is None:\n            is_path = isinstance(source, (str, Path)) and Path(str(source)).exists()\n        if is_path:\n            data = Path(str(source)).read_bytes()\n        else:\n            data = str(source).encode(\"utf-8\", \"replace\")\n        return \"sha256:\" + hashlib.sha256(data).hexdigest()[:12]\n    except Exception:\n        return \"sha256:UNREADABLE\"\n\n\ndef read_label_corpus() -> dict:\n    \"\"\"Read the owner's claude-reflect corrections queue = kappa ground-truth labels.\n    Caught-error corpus (blind to escaped defects) -> licenses SPEAK, never BLOCK (C3).\"\"\"\n    if not LABEL_CORPUS.exists():\n        return {\"present\": False, \"n\": 0, \"reason\": \"learnings-queue.json absent (no corrections captured yet)\"}\n    try:\n        items = json.loads(LABEL_CORPUS.read_text(encoding=\"utf-8\"))\n        corrections = [it for it in items if isinstance(it, dict)\n                       and (it.get(\"sentiment\") == \"correction\" or it.get(\"type\") in (\"auto\", \"explicit\", \"guardrail\"))]\n        return {\"present\": True, \"n\": len(corrections), \"total\": len(items)}\n    except Exception as e:\n        return {\"present\": False, \"n\": 0, \"reason\": f\"unreadable: {e!r}\"}\n\n\ndef probe_passk_stats() -> dict:\n    \"\"\"Detect the F5 pass^k \u002F e-value \u002F kappa calibrator. Absent -> advisory, never fabricate.\"\"\"\n    # F5 may ship as a skill dir, a module, or an artifact. Probe the known shapes.\n    candidates = [\n        HOME \u002F \".claude\" \u002F \"skills\" \u002F \"passk-stats\" \u002F \"SKILL.md\",\n        HOME \u002F \".claude\" \u002F \"skills\" \u002F \"pass-k-stats\" \u002F \"SKILL.md\",\n        HOME \u002F \".claude\" \u002F \"hooks\" \u002F \"passk_stats.py\",\n    ]\n    for c in candidates:\n        if c.exists():\n            return {\"present\": True, \"path\": str(c)}\n    try:\n        import passk_stats  # noqa: F401,PLC0415\n        return {\"present\": True, \"path\": \"module:passk_stats\"}\n    except Exception:\n        return {\"present\": False}\n\n\ndef calibration_status(labels: dict, passk: dict) -> str:\n    \"\"\"The honest, non-fabricating calibration verdict. Never returns a made-up kappa.\"\"\"\n    if not passk.get(\"present\"):\n        return \"CALIBRATION: UNAVAILABLE (passk-stats\u002FF5 not present) -> advisory\"\n    if not labels.get(\"present\") or labels.get(\"n\", 0) \u003C MIN_LABELED:\n        return (f\"CALIBRATION: INSUFFICIENT LABELS (n={labels.get('n', 0)} \u003C {MIN_LABELED}) \"\n                f\"-> advisory (honest prior kappa \u003C {KAPPA_BAR})\")\n    # F5 present AND enough labels: hand off to passk-stats to compute per-category kappa + e-value.\n    # This helper does NOT compute kappa itself (that is F5's job); it reports the handoff.\n    return (f\"CALIBRATION: DELEGATE to passk-stats over {labels['n']} labels \"\n            f\"(per-category kappa, lower-CI-bound gate, bar={KAPPA_BAR}); stays advisory until owner promotes\")\n\n\ndef shadow_log(record: dict) -> str:\n    \"\"\"Append ONE JSON line to the bounded, rotating shadow sink. Never the main ledger.\"\"\"\n    try:\n        SINK.parent.mkdir(parents=True, exist_ok=True)\n        if SINK.exists() and SINK.stat().st_size > SINK_ROTATE_BYTES:\n            rotated = SINK.with_suffix(SINK.suffix + \".1\")\n            try:\n                if rotated.exists():\n                    rotated.unlink()\n                SINK.rename(rotated)\n            except Exception:\n                pass\n        record = {\"ts\": time.strftime(\"%Y-%m-%dT%H:%M:%S\"), \"mode\": \"SHADOW\u002FADVISE\",\n                  \"blocks\": False, **record}\n        with SINK.open(\"a\", encoding=\"utf-8\") as fh:\n            fh.write(json.dumps(record, ensure_ascii=False) + \"\\n\")\n        return str(SINK)\n    except Exception as e:\n        return f\"SHADOW-LOG-FAILED: {e!r}\"\n\n\ndef selftest() -> int:\n    \"\"\"Run against the LIVE ledger + label corpus; emit a demo advisory verdict. Blocks nothing.\"\"\"\n    project = os.environ.get(\"CLAUDE_CWD\", os.getcwd())\n    events = cite_ledger(project, n=10)\n    labels = read_label_corpus()\n    passk = probe_passk_stats()\n    cal = calibration_status(labels, passk)\n\n    # Demo DoD item cited two legal ways (ledger event if any exist, else an evidence hash).\n    if events:\n        item_cite = ledger_citation(events[-1])\n        cite_kind = \"ledger-event\"\n    else:\n        item_cite = hash_evidence(\"DEMO: no ledger events yet for this project\", is_path=False)\n        cite_kind = \"evidence-hash (no ledger events -> hashed the observation)\"\n\n    verdict = {\n        \"verdict\": \"ADVISORY (does not gate)\",\n        \"ran_after_deterministic_cluster\": \"unknown (selftest)\",\n        \"checklist_demo\": [{\n            \"item\": \"DoD item extracted (demo)\", \"type\": \"subjective\", \"opinion\": \"YES\",\n            \"citation\": item_cite, \"citation_kind\": cite_kind, \"receipt\": \"unrunnable (demo)\",\n        }],\n        \"icd203\": \"9-standard rubric applied per item (see agents\u002Fagent-as-judge.md)\",\n        \"calibration\": cal,\n        \"ledger_events_available_for_citation\": len(events),\n        \"label_corpus\": labels,\n        \"passk_stats\": passk,\n        \"blocking_findings\": 0,\n    }\n    sink = shadow_log({\"event\": \"selftest\", \"verdict\": verdict})\n    print(\"=== Agent-as-a-Judge selftest (ADVISORY — blocks nothing) ===\")\n    print(json.dumps(verdict, ensure_ascii=False, indent=2))\n    print(f\"SHADOW-LOG -> {sink}\")\n    print(\"SUCCESS:agent-as-judge selftest — advisory verdict emitted, 0 blocking findings\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(selftest() if \"--selftest\" in sys.argv else selftest())\n```\n\n## Honest limits\n\n- **Not a done-signal.** By council law it cannot block or clear; the deterministic cluster owns\n  the done-signal. If someone wires this into a blocking path, that violates C3.\n- **Correlated failure.** A same-vendor judge shares the fixer's blind spots; a cross-vendor Codex\n  second judge is the independent check for contested\u002Firreversible completions.\n- **Calibration corpus is caught-error-only.** kappa vs claude-reflect corrections optimizes for\n  re-catching *known* error classes; it is blind to escaped defects, so it can never license\n  opinion-gating (only a fresh prospective held-out set the judge never saw could — deferred).\n- **F5 dependency.** kappa\u002Fe-value calibration requires `passk-stats`; until it exists the judge is\n  advisory with `CALIBRATION: UNAVAILABLE`. That is the correct, safe default — not a bug.\n",{"name":5,"description":7},"main","a298c929b6cbd662e1fce54302ce520d5ceea1a5","skills\u002Fagent-as-judge","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain","https:\u002F\u002Fgithub.com\u002FPrekzursil","2026-11-02 14:30:57",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add Prekzursil\u002Fagent-skills-toolchain --skill agent-as-judge",{"tool":60,"command":61},"bunx","bunx skills add Prekzursil\u002Fagent-skills-toolchain --skill agent-as-judge",{"tool":63,"command":64},"pnpm","pnpm dlx skills add Prekzursil\u002Fagent-skills-toolchain --skill agent-as-judge","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Fskills\u002Fagent-as-judge\u002FSKILL.md",true,[],[69,89,116,143,157,173],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":43},"01M6DF7D7478R55P46REYH4QAC","clarify-requirements","Resolve genuine pre-execution ambiguity by scanning repo context, generating 1-6 grounded options, and asking 1-3 focused questions via AskUserQuestion. NOT for mid-task use (injection risk); NOT when the goal is already unambiguous; use to disambiguate vague prompts before any file edits.","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Ftools\u002Fworkspace-hooks\u002Fskills\u002Fclarify-requirements\u002FSKILL.md",[],[76,77,78,79,80,81,82,83,84,35,85,86,87,39,88],"clarify requirements","clarify","requirements","risk","options","task","prompts","goal","scanning","context","repo","injection","file",{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":96,"publishedAt":43},"01M6DF79CA4BHJS66J8EZDBPWJ","openmm-md","Run FREE, open-source, Windows-native GPU molecular dynamics via the openmm-md MCP (OpenMM + CUDA on the local NVIDIA GPU). Use for protein \u002F protein-ligand MD when you want a free stack or are on Windows without a Schrodinger license. Covers the full workflow (prepare, solvate, minimize, equilibrate, production, analyze, mmgbsa) AND an honest decision guide vs Schrodinger Desmond, including how to run Desmond legitimately on Linux or a cloud GPU with a real license. This is a deliberately LESSER free alternative and does NOT replace the schrodinger MCP tools. Do NOT use it to pick a free tool when a validated FEP+, WaterMap, IFD-MD, or Prime MM-GBSA result is actually required.","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Ftools\u002Fmcp\u002Fopenmm-md-mcp\u002Fskill\u002Fopenmm-md\u002FSKILL.md",[95],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[97,98,99,100,101,102,103,104,105,106,107,108,109,110,35,111,112,113,114,115],"openmm md","how","decision","pick","linux","analyze","tool","stack","open","NVIDIA","gpu","license","workflow","windows","free","molecular dynamics","local","do","MCP",{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":128,"publishedAt":43},"01M6DF76C1MG85VEWP8VZCBNMN","visual-review","Take screenshots of web pages and UI using Playwright for visual review and iteration","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Ftools\u002Feditor\u002Fmetaswarm\u002Fskills\u002Fvisual-review\u002FSKILL.md",[122,125],{"categoryNo":123,"categoryName":124,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000003","演示文稿",{"categoryNo":126,"categoryName":127,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",[129,130,131,132,133,134,135,136,137,138,98,139,140,141,142],"review","visual","visual review","pages","web","false","iteration","take","check","slides","ui","screenshot","playwright","screenshots",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":43},"01M6DF73AQVCBZKEN0YXEX9FXP","status","Diagnostic status report — shows metaswarm installation state, project setup, and potential issues","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Ftools\u002Feditor\u002Fmetaswarm\u002Fskills\u002Fstatus\u002FSKILL.md",[],[145,150,151,152,153,154,155,156],"setup","report","project","installation","project setup","issues","state",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":164,"publishedAt":43},"01M6DF6X6C2XB0P6GX76P7TTYV","start","Use when starting work on any task, when the user mentions metaswarm, or when the user wants to begin tracked development work","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Ftools\u002Feditor\u002Fmetaswarm\u002Fskills\u002Fstart\u002FSKILL.md",[163],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[159,165,81,166,167,168,35,169,170,171,172],"start issue","issue","ready","work on issue","user","work","work on","development",{"skillNo":174,"name":150,"nameZh":6,"description":175,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":176,"categories":177,"keywords":178,"publishedAt":43},"01M6DF6QMJ9AJKFKHHKRT005PQ","Interactive project setup — detects your project, configures metaswarm, writes project-local files","https:\u002F\u002Fgithub.com\u002FPrekzursil\u002Fagent-skills-toolchain\u002Fblob\u002Fmain\u002Ftools\u002Feditor\u002Fmetaswarm\u002Fskills\u002Fsetup\u002FSKILL.md",[],[150,152,113,179,154,180],"files","interactive"]