[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7572KGQKE6MCBAQJS713443":3,"skill-same:01M7572KGQKE6MCBAQJS713443":88},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M7572KGQKE6MCBAQJS713443","llm-as-verifier",null,"This skill should be used when the user asks to \"compare candidate patches\", \"rank multiple solutions\", \"choose between implementations\", \"verify two drafts\", \"pick the best trajectory\", \"use llm-as-a-verifier\", or wants repeated criteria-decomposed LLM verification instead of a single free-form judgment.","kingkillery\u002Foh-my-pk","kingkillery","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F200727508?v=4","01M6X59E9Q2FCEGBVQ1P6FT0YC","PK","01M756YT55MBWAW7Q3BJ768ADB",1,"MIT","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: llm-as-verifier\ndescription: This skill should be used when the user asks to \"compare candidate patches\", \"rank multiple solutions\", \"choose between implementations\", \"verify two drafts\", \"pick the best trajectory\", \"use llm-as-a-verifier\", or wants repeated criteria-decomposed LLM verification instead of a single free-form judgment.\n---\n\n# LLM-as-Verifier\n\n## Purpose\n\nUse this skill to choose among multiple candidate solutions with a structured verifier loop inspired by the `llm-as-a-verifier` paper and repo.\n\nPrefer this workflow when:\n- a single model judgment would be too brittle,\n- several drafts, patches, or plans already exist,\n- the task benefits from explicit evaluation criteria,\n- deterministic evidence exists and should be weighed more heavily than narration.\n\nDo **not** treat the verifier as a replacement for tests, logs, or direct inspection. It is a selection mechanism layered on top of concrete evidence.\n\n## Core idea\n\nReplace one-shot \"which answer is best?\" judging with four stronger moves:\n\n1. **Compare candidates pairwise** instead of scoring only one artifact in isolation.\n2. **Decompose judgment into criteria** such as correctness, requirements adherence, empirical verification, or clarity.\n3. **Repeat the verification** several times to reduce single-sample noise.\n4. **Use a tournament\u002Franking pass** to select the overall winner across candidates.\n\n## Prefer compare mode over audit mode\n\nUse **compare** mode whenever at least two viable candidates exist. Pairwise comparison is the closest match to the paper and usually produces a stronger signal than absolute scoring.\n\nUse **audit** mode only when a single artifact must be scored against explicit criteria and no alternative candidate exists yet.\n\n## Inputs\n\nProvide as many of these as the task supports:\n\n- `task`\n- `candidates`\n- `criteria`\n- optional shared `context`\n- optional shared `evidencePaths`\n- optional `nVerifications`\n- optional `outputPath`\n- optional `backend`\n- optional `models`\n- optional `modelWeights`\n\nFor each candidate, prefer:\n- stable `id`\n- focused `content` or file `path`\n- optional short `summary`\n- optional candidate-specific evidence\n\n## Output behavior\n\nA successful verifier run should produce:\n- a winner or audit score,\n- criterion-by-criterion reasoning structure,\n- repeated verification results,\n- confidence and disagreement signals,\n- machine-readable JSON output,\n- a final explanation of why the result won.\n\nWhen confidence is low or evidence is weak, say so explicitly and recommend the next deterministic check.\n\n## Backends and model lanes in this packet\n\nThree verifier backends are available:\n\n- `gemini-python` — uses the deterministic Python runner in `scripts\u002Flav_runner.py`\n- `zai-coding-plan` — runs the verifier flow through a single Pi-configured ZAI coding model\n- `pi-model-ensemble` — rotates repeated verification attempts across multiple Pi-configured models\n\nDefault ensemble lane selection is aligned with the current `\u002Fdelegate` setup:\n\n- `kimi:kimi-for-coding` — prompt\u002Frequirements-sensitive verifier lane\n- `minimax.io:minimax-m3` — independent MiniMax M3 verifier lane\n- `openai:gpt-5.5` — strong OpenAI\u002FCodex-side verifier and synthesis lane\n\nWhen verifying outputs produced by `\u002Fdelegate`, treat the lane artifacts as candidates or evidence:\n\n- Kimi-refined prompt: shared context\u002Fevidence, not a candidate unless evaluating prompt quality.\n- Minimax lane output: candidate or candidate-specific evidence.\n- OpenAI Codex lane output: candidate or candidate-specific evidence.\n- GPT-5.5 via Codex synthesis: audit target or candidate only if comparing synthesized reports.\n\nThe ensemble backend also supports:\n- per-model weighting through `modelWeights`\n- confidence and disagreement reporting\n- per-model breakdowns across repeated attempts\n\nAll three backends perform:\n- repeated per-criterion scoring\n- pairwise aggregation\n- round-robin selection or audit averaging\n- machine-readable JSON output\n\n### Interactive selection menu\n\nCreate an interactive menu only when the user has not specified a backend\u002Fmodel set and the choice has material tradeoffs:\n\n1. **Fast deterministic smoke** — `gemini-python` with `mock: true`; use only for pipeline validation.\n2. **Default compare** — `pi-model-ensemble` with `kimi:kimi-for-coding`, `minimax.io:minimax-m3`, and `openai:gpt-5.5`; recommended for normal multi-candidate selection.\n3. **Single strong verifier** — `zai-coding-plan` with `zai:glm-5.1`; use when ensemble auth is unavailable or the user wants one Pi-routed model.\n4. **Delegate-output review** — `pi-model-ensemble` with `\u002Fdelegate` lane outputs supplied as candidates\u002Fevidence; use after `\u002Fdelegate` generates Minimax\u002FCodex lane artifacts.\n\nIf the safest default is clear, do not ask: choose **Default compare** for 2+ candidates and **Single strong verifier** only for a single-candidate audit.\n\n## Required process\n\n### 1. Gather deterministic evidence first\n\nCollect the strongest available evidence before calling the verifier:\n\n- build\u002Ftest output\n- terminal logs\n- diff or patch text\n- spec excerpts\n- generated files\n- failing vs passing examples\n\nPrefer observed outputs over self-reported claims from the candidate.\n\n### 2. Define 3-5 criteria\n\nWrite criteria that isolate different failure modes. Keep each criterion narrow and operational.\n\nGood examples:\n- **Correctness** — Does the artifact actually solve the requested problem?\n- **Requirements adherence** — Does it match exact constraints, file paths, formats, and scope?\n- **Empirical verification** — Does the evidence show the fix was tested and validated?\n- **Maintainability** — Does the change fit surrounding code and avoid obvious regressions?\n\nAvoid overlapping criteria like \"quality\", \"goodness\", and \"correctness\" all at once. That creates duplicate scoring.\n\nUse `references\u002Fcriteria-recipes.md` for templates.\n\n### 3. Prepare candidate payloads\n\nKeep candidate payloads focused. Include the patch, proposal, response, or document section that actually needs comparison. Trim unrelated noise.\n\n### 4. Run the verifier\n\nUse the project-local `llm_as_verifier` extension tool when available. It is auto-registered from:\n- `.pi\u002Fextensions\u002Fllm-as-verifier\u002Findex.ts`\n\n### 5. Sanity-check the winner\n\nDo not stop at the selected winner. Verify that the result agrees with deterministic evidence and user constraints.\n\nIf the winner contradicts tests or explicit requirements:\n- inspect the pairwise breakdown,\n- tighten the criteria,\n- add missing evidence,\n- rerun compare mode.\n\n### 6. Explain the decision\n\nSummarize:\n- why the winner won,\n- which criteria decided the outcome,\n- what evidence mattered most,\n- any residual uncertainty.\n\n## Setup\n\nFor `gemini-python` real runs, install the Gemini client:\n\n```bash\npip install google-genai\n```\n\nProvide one of these environment variables or a local `.env` file:\n\n- `GEMINI_API_KEY`\n- `GOOGLE_API_KEY`\n- `VERTEX_API_KEY`\n\nFor `zai-coding-plan` real runs, configure ZAI auth in Pi so the extension can access a ZAI model such as `glm-5.1` through Pi's model registry.\n\nFor `pi-model-ensemble` real runs, configure auth in Pi for each provider\u002Fmodel you want to rotate across. The default verifier panel is:\n\n- `kimi:kimi-for-coding`\n- `minimax.io:minimax-m3`\n- `openai:gpt-5.5` (GPT-5.5 via the OpenAI\u002FCodex side)\n\nOther supported examples:\n\n- `minimax:MiniMax-M2.7-highspeed`\n- `openai:gpt-5-codex`\n- `openai:gpt-5.4`\n- `google:gemini-2.5-flash`\n\nUse `mock` mode only for smoke tests and pipeline validation.\n\n## Tooling in this packet\n\n### Settings and slash command\n\nUse `\u002Fsettings` → **Tasks** → **Subagents** → **LLM Verifier Model** to choose the model used by the verifier lane. Prefer a model\u002Fprovider independent from the implementation model.\n\nRun `\u002Fverify` after implementation to audit the current staged and unstaged tracked diff. With no arguments, the command opens an objective\u002Facceptance editor. Inline form:\n\n```text\n\u002Fverify Objective: preserve installer defaults\n\nAcceptance:\n- npm\u002FBun remains the default\n- unsupported binary architectures fail before download\n- focused installer tests pass\n```\n\n`\u002Fverify` runs a repeated single-candidate audit and reports criterion-level evidence. It does not run tests for the implementer. The command audits tracked diffs only; use `llm_as_verifier` directly for saved or untracked candidates and multi-candidate comparison.\n\nThe command fails before contacting the model when the tracked diff exceeds 40,000 characters or status evidence exceeds 20,000 characters, so verification never silently scores a truncated workspace. Narrow the change set or call `llm_as_verifier` with a saved candidate.\n\n### Extension tool\n\nUse the `llm_as_verifier` tool for normal operation.\n\nThe tool supports:\n- `compare` mode for 2-6 candidates\n- `audit` mode for 1 candidate\n- `backend` selection: `gemini-python`, `zai-coding-plan`, or `pi-model-ensemble`\n- `models` for ensemble rotation across repeated attempts; known aliases include `kimi-for-coding`, `kimi-k2` (mapped to `kimi-for-coding`), `minimax-m3`, `minimax-m2.7-highspeed`, `gpt-5.5`, and `gpt-5-codex`\n- `modelWeights` for weighted ensemble aggregation\n- confidence\u002Fdisagreement reporting in result details\n- `mock` mode for smoke tests only\n\n### Deterministic runner\n\nUse `scripts\u002Flav_runner.py` directly when the extension is unavailable or when a machine-readable batch run is preferred.\n\nExample:\n\n```bash\npython .agents\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Flav_runner.py \\\n  --input \u002Fpath\u002Fto\u002Finput.json \\\n  --output \u002Fpath\u002Fto\u002Fresult.json\n```\n\n### Smoke commands\n\nUse the bundled smoke commands before higher-stakes runs:\n\n```bash\n\u002Flav-smoke\n\u002Flav-ensemble-smoke\n```\n\n- `\u002Flav-smoke` runs the bundled Python-runner example in deterministic mock mode.\n- `\u002Flav-ensemble-smoke` runs the bundled weighted ensemble example in deterministic mock mode and exercises model rotation plus weighted aggregation.\n\n## Rules\n\n- Prefer **compare** mode over **audit** mode whenever there are multiple viable candidates.\n- Gather deterministic evidence before asking the verifier to rank anything.\n- Keep criteria narrow, operational, and non-overlapping.\n- Keep candidate payloads focused; trim unrelated noise.\n- Treat strong evidence as more trustworthy than polished narration.\n- When uncertainty is high, say so and recommend a next deterministic check.\n- Do not present verifier output as certainty when the inputs are weak.\n- Use mock mode only for smoke tests, demos, and pipeline validation.\n- For `\u002Fdelegate` results, preserve separate lane outputs. Do not collapse Minimax, Codex, and GPT-5.5 synthesis into one candidate unless the question is about the combined final response.\n\n## Criteria-writing rules\n\nUse this pattern:\n- **Name**: short label\n- **Description**: exact thing to inspect, what counts as strong evidence, what counts as failure\n\nPrefer wording like:\n- \"Check whether the final patch edits the actual failing code path rather than a downstream symptom.\"\n- \"Compare the observed output to the required output and score only the match between them.\"\n- \"Reward candidates whose final artifact is supported by concrete test evidence.\"\n\nAvoid vague wording like:\n- \"Judge overall quality.\"\n- \"See whether it seems better.\"\n- \"Pick the one you like most.\"\n\n## Failure modes to watch\n\n- Missing evidence causes the verifier to over-trust polished narration.\n- Overlapping criteria double-count the same weakness.\n- Huge candidate payloads bury the important signal.\n- Audit mode on a single candidate is weaker than compare mode.\n- Strong style can beat weak correctness unless criteria explicitly protect correctness.\n\n## Additional resources\n\nRead these references when shaping a verifier run:\n\n- `references\u002Fresearch-notes.md` — paper-to-implementation mapping and design rationale\n- `references\u002Fcriteria-recipes.md` — reusable criteria sets for code, plans, docs, and answers\n- `examples\u002Fcode-patch-selection.json` — example compare input for the Python runner and `\u002Flav-smoke`\n- `examples\u002Fweighted-ensemble-selection.json` — weighted ensemble example for `\u002Flav-ensemble-smoke`\n\nPrompt templates are also bundled for quick starts:\n\n- `\u002Fcompare-patches`\n- `\u002Faudit-candidate`\n- `\u002Fensemble-verifier`\n\n## Success criteria\n\nThis skill is successful when it:\n- chooses the strongest candidate rather than the most polished one,\n- makes the decision legible through explicit criteria,\n- incorporates deterministic evidence into the ranking,\n- surfaces uncertainty honestly,\n- produces a machine-readable artifact when needed,\n- helps the user rerun the comparison with sharper inputs when the result is weak.\n\n## Operating guidance\n\nStart with compare mode.\n\nUse **5 verifier samples** as the default for normal work. The Python runner\n(`scripts\u002Flav_runner.py`) defaults `n_verifications` to 5; the Pi extension\ndefaults `nVerifications` to `max(5, models.length)` for `pi-model-ensemble`\nand 5 for `gemini-python` \u002F `zai-coding-plan`. Explicit lower values\n(`n_verifications` in `1..8`) are still accepted for smoke tests, but treat\nanything below 5 as a high-variance decision aid, not a verdict.\n\n**Compare mode always runs A\u002FB and B\u002FA orderings.** Every pair \u002F criterion \u002F\nrepetition evaluates both `candidate_a → candidate_b` (original) and\n`candidate_b → candidate_a` (swapped). The canonical score for the swapped\nrun is fed back into the aggregate, so a position-only bias cannot pick a\nwinner on its own.\n\n**Low vote margin returns `tie` \u002F uncertain.** Pairwise compare mode counts\ncanonical per-repetition winners (ties count `0.5` for each candidate) and\ncomputes `vote_margin = max(votes) \u002F total_votes`. When `vote_margin \u003C 0.7`\nthe pair winner is forced to `tie` even if the mean scores differ — the\nnumeric scores stay visible, but the categorical decision is `uncertain`.\nAudit mode applies the same rule to its positive\u002Fnegative vote tally.\n\n**Candidate outputs are untrusted and scanned.** Every candidate answer is\nrouted through `scan_for_judge_manipulation`\n(`harness\u002Fsecurity\u002Fprompt_injection.py`) before rubric scoring. Flagged\ncandidates receive `judge-manipulation: \u003Cpattern>` weaknesses (penalizing\n`evidence_quality` and `safety_permission_fit`) and the run surfaces a\ntop-level warning in the format\n`candidate \u003Cid> contains judge-manipulation patterns: \u003Cflags>`. Legitimate\nsecurity discussion may be flagged and stays advisory plus penalty, not\nhard failure.\n\nIn `pi-model-ensemble`, if `nVerifications` is omitted, it defaults to\n`max(5, models.length)` so each model still gets at least one pass. The\ndefault panel is Kimi, MiniMax M3 from `minimax.io`, and GPT-5.5 via\nOpenAI\u002FCodex. Increase verifications only when the decision is high value\nand the candidate set is small.\n\nSupply shared evidence whenever the task has external ground truth.\n\nUse audit mode only as a fallback.\n\nTreat the verifier as a structured selector layered on top of deterministic evidence, not as a magical replacement for tests.\n","# LLM-as-Verifier\n\n## Purpose\n\nUse this skill to choose among multiple candidate solutions with a structured verifier loop inspired by the `llm-as-a-verifier` paper and repo.\n\nPrefer this workflow when:\n- a single model judgment would be too brittle,\n- several drafts, patches, or plans already exist,\n- the task benefits from explicit evaluation criteria,\n- deterministic evidence exists and should be weighed more heavily than narration.\n\nDo **not** treat the verifier as a replacement for tests, logs, or direct inspection. It is a selection mechanism layered on top of concrete evidence.\n\n## Core idea\n\nReplace one-shot \"which answer is best?\" judging with four stronger moves:\n\n1. **Compare candidates pairwise** instead of scoring only one artifact in isolation.\n2. **Decompose judgment into criteria** such as correctness, requirements adherence, empirical verification, or clarity.\n3. **Repeat the verification** several times to reduce single-sample noise.\n4. **Use a tournament\u002Franking pass** to select the overall winner across candidates.\n\n## Prefer compare mode over audit mode\n\nUse **compare** mode whenever at least two viable candidates exist. Pairwise comparison is the closest match to the paper and usually produces a stronger signal than absolute scoring.\n\nUse **audit** mode only when a single artifact must be scored against explicit criteria and no alternative candidate exists yet.\n\n## Inputs\n\nProvide as many of these as the task supports:\n\n- `task`\n- `candidates`\n- `criteria`\n- optional shared `context`\n- optional shared `evidencePaths`\n- optional `nVerifications`\n- optional `outputPath`\n- optional `backend`\n- optional `models`\n- optional `modelWeights`\n\nFor each candidate, prefer:\n- stable `id`\n- focused `content` or file `path`\n- optional short `summary`\n- optional candidate-specific evidence\n\n## Output behavior\n\nA successful verifier run should produce:\n- a winner or audit score,\n- criterion-by-criterion reasoning structure,\n- repeated verification results,\n- confidence and disagreement signals,\n- machine-readable JSON output,\n- a final explanation of why the result won.\n\nWhen confidence is low or evidence is weak, say so explicitly and recommend the next deterministic check.\n\n## Backends and model lanes in this packet\n\nThree verifier backends are available:\n\n- `gemini-python` — uses the deterministic Python runner in `scripts\u002Flav_runner.py`\n- `zai-coding-plan` — runs the verifier flow through a single Pi-configured ZAI coding model\n- `pi-model-ensemble` — rotates repeated verification attempts across multiple Pi-configured models\n\nDefault ensemble lane selection is aligned with the current `\u002Fdelegate` setup:\n\n- `kimi:kimi-for-coding` — prompt\u002Frequirements-sensitive verifier lane\n- `minimax.io:minimax-m3` — independent MiniMax M3 verifier lane\n- `openai:gpt-5.5` — strong OpenAI\u002FCodex-side verifier and synthesis lane\n\nWhen verifying outputs produced by `\u002Fdelegate`, treat the lane artifacts as candidates or evidence:\n\n- Kimi-refined prompt: shared context\u002Fevidence, not a candidate unless evaluating prompt quality.\n- Minimax lane output: candidate or candidate-specific evidence.\n- OpenAI Codex lane output: candidate or candidate-specific evidence.\n- GPT-5.5 via Codex synthesis: audit target or candidate only if comparing synthesized reports.\n\nThe ensemble backend also supports:\n- per-model weighting through `modelWeights`\n- confidence and disagreement reporting\n- per-model breakdowns across repeated attempts\n\nAll three backends perform:\n- repeated per-criterion scoring\n- pairwise aggregation\n- round-robin selection or audit averaging\n- machine-readable JSON output\n\n### Interactive selection menu\n\nCreate an interactive menu only when the user has not specified a backend\u002Fmodel set and the choice has material tradeoffs:\n\n1. **Fast deterministic smoke** — `gemini-python` with `mock: true`; use only for pipeline validation.\n2. **Default compare** — `pi-model-ensemble` with `kimi:kimi-for-coding`, `minimax.io:minimax-m3`, and `openai:gpt-5.5`; recommended for normal multi-candidate selection.\n3. **Single strong verifier** — `zai-coding-plan` with `zai:glm-5.1`; use when ensemble auth is unavailable or the user wants one Pi-routed model.\n4. **Delegate-output review** — `pi-model-ensemble` with `\u002Fdelegate` lane outputs supplied as candidates\u002Fevidence; use after `\u002Fdelegate` generates Minimax\u002FCodex lane artifacts.\n\nIf the safest default is clear, do not ask: choose **Default compare** for 2+ candidates and **Single strong verifier** only for a single-candidate audit.\n\n## Required process\n\n### 1. Gather deterministic evidence first\n\nCollect the strongest available evidence before calling the verifier:\n\n- build\u002Ftest output\n- terminal logs\n- diff or patch text\n- spec excerpts\n- generated files\n- failing vs passing examples\n\nPrefer observed outputs over self-reported claims from the candidate.\n\n### 2. Define 3-5 criteria\n\nWrite criteria that isolate different failure modes. Keep each criterion narrow and operational.\n\nGood examples:\n- **Correctness** — Does the artifact actually solve the requested problem?\n- **Requirements adherence** — Does it match exact constraints, file paths, formats, and scope?\n- **Empirical verification** — Does the evidence show the fix was tested and validated?\n- **Maintainability** — Does the change fit surrounding code and avoid obvious regressions?\n\nAvoid overlapping criteria like \"quality\", \"goodness\", and \"correctness\" all at once. That creates duplicate scoring.\n\nUse `references\u002Fcriteria-recipes.md` for templates.\n\n### 3. Prepare candidate payloads\n\nKeep candidate payloads focused. Include the patch, proposal, response, or document section that actually needs comparison. Trim unrelated noise.\n\n### 4. Run the verifier\n\nUse the project-local `llm_as_verifier` extension tool when available. It is auto-registered from:\n- `.pi\u002Fextensions\u002Fllm-as-verifier\u002Findex.ts`\n\n### 5. Sanity-check the winner\n\nDo not stop at the selected winner. Verify that the result agrees with deterministic evidence and user constraints.\n\nIf the winner contradicts tests or explicit requirements:\n- inspect the pairwise breakdown,\n- tighten the criteria,\n- add missing evidence,\n- rerun compare mode.\n\n### 6. Explain the decision\n\nSummarize:\n- why the winner won,\n- which criteria decided the outcome,\n- what evidence mattered most,\n- any residual uncertainty.\n\n## Setup\n\nFor `gemini-python` real runs, install the Gemini client:\n\n```bash\npip install google-genai\n```\n\nProvide one of these environment variables or a local `.env` file:\n\n- `GEMINI_API_KEY`\n- `GOOGLE_API_KEY`\n- `VERTEX_API_KEY`\n\nFor `zai-coding-plan` real runs, configure ZAI auth in Pi so the extension can access a ZAI model such as `glm-5.1` through Pi's model registry.\n\nFor `pi-model-ensemble` real runs, configure auth in Pi for each provider\u002Fmodel you want to rotate across. The default verifier panel is:\n\n- `kimi:kimi-for-coding`\n- `minimax.io:minimax-m3`\n- `openai:gpt-5.5` (GPT-5.5 via the OpenAI\u002FCodex side)\n\nOther supported examples:\n\n- `minimax:MiniMax-M2.7-highspeed`\n- `openai:gpt-5-codex`\n- `openai:gpt-5.4`\n- `google:gemini-2.5-flash`\n\nUse `mock` mode only for smoke tests and pipeline validation.\n\n## Tooling in this packet\n\n### Settings and slash command\n\nUse `\u002Fsettings` → **Tasks** → **Subagents** → **LLM Verifier Model** to choose the model used by the verifier lane. Prefer a model\u002Fprovider independent from the implementation model.\n\nRun `\u002Fverify` after implementation to audit the current staged and unstaged tracked diff. With no arguments, the command opens an objective\u002Facceptance editor. Inline form:\n\n```text\n\u002Fverify Objective: preserve installer defaults\n\nAcceptance:\n- npm\u002FBun remains the default\n- unsupported binary architectures fail before download\n- focused installer tests pass\n```\n\n`\u002Fverify` runs a repeated single-candidate audit and reports criterion-level evidence. It does not run tests for the implementer. The command audits tracked diffs only; use `llm_as_verifier` directly for saved or untracked candidates and multi-candidate comparison.\n\nThe command fails before contacting the model when the tracked diff exceeds 40,000 characters or status evidence exceeds 20,000 characters, so verification never silently scores a truncated workspace. Narrow the change set or call `llm_as_verifier` with a saved candidate.\n\n### Extension tool\n\nUse the `llm_as_verifier` tool for normal operation.\n\nThe tool supports:\n- `compare` mode for 2-6 candidates\n- `audit` mode for 1 candidate\n- `backend` selection: `gemini-python`, `zai-coding-plan`, or `pi-model-ensemble`\n- `models` for ensemble rotation across repeated attempts; known aliases include `kimi-for-coding`, `kimi-k2` (mapped to `kimi-for-coding`), `minimax-m3`, `minimax-m2.7-highspeed`, `gpt-5.5`, and `gpt-5-codex`\n- `modelWeights` for weighted ensemble aggregation\n- confidence\u002Fdisagreement reporting in result details\n- `mock` mode for smoke tests only\n\n### Deterministic runner\n\nUse `scripts\u002Flav_runner.py` directly when the extension is unavailable or when a machine-readable batch run is preferred.\n\nExample:\n\n```bash\npython .agents\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Flav_runner.py \\\n  --input \u002Fpath\u002Fto\u002Finput.json \\\n  --output \u002Fpath\u002Fto\u002Fresult.json\n```\n\n### Smoke commands\n\nUse the bundled smoke commands before higher-stakes runs:\n\n```bash\n\u002Flav-smoke\n\u002Flav-ensemble-smoke\n```\n\n- `\u002Flav-smoke` runs the bundled Python-runner example in deterministic mock mode.\n- `\u002Flav-ensemble-smoke` runs the bundled weighted ensemble example in deterministic mock mode and exercises model rotation plus weighted aggregation.\n\n## Rules\n\n- Prefer **compare** mode over **audit** mode whenever there are multiple viable candidates.\n- Gather deterministic evidence before asking the verifier to rank anything.\n- Keep criteria narrow, operational, and non-overlapping.\n- Keep candidate payloads focused; trim unrelated noise.\n- Treat strong evidence as more trustworthy than polished narration.\n- When uncertainty is high, say so and recommend a next deterministic check.\n- Do not present verifier output as certainty when the inputs are weak.\n- Use mock mode only for smoke tests, demos, and pipeline validation.\n- For `\u002Fdelegate` results, preserve separate lane outputs. Do not collapse Minimax, Codex, and GPT-5.5 synthesis into one candidate unless the question is about the combined final response.\n\n## Criteria-writing rules\n\nUse this pattern:\n- **Name**: short label\n- **Description**: exact thing to inspect, what counts as strong evidence, what counts as failure\n\nPrefer wording like:\n- \"Check whether the final patch edits the actual failing code path rather than a downstream symptom.\"\n- \"Compare the observed output to the required output and score only the match between them.\"\n- \"Reward candidates whose final artifact is supported by concrete test evidence.\"\n\nAvoid vague wording like:\n- \"Judge overall quality.\"\n- \"See whether it seems better.\"\n- \"Pick the one you like most.\"\n\n## Failure modes to watch\n\n- Missing evidence causes the verifier to over-trust polished narration.\n- Overlapping criteria double-count the same weakness.\n- Huge candidate payloads bury the important signal.\n- Audit mode on a single candidate is weaker than compare mode.\n- Strong style can beat weak correctness unless criteria explicitly protect correctness.\n\n## Additional resources\n\nRead these references when shaping a verifier run:\n\n- `references\u002Fresearch-notes.md` — paper-to-implementation mapping and design rationale\n- `references\u002Fcriteria-recipes.md` — reusable criteria sets for code, plans, docs, and answers\n- `examples\u002Fcode-patch-selection.json` — example compare input for the Python runner and `\u002Flav-smoke`\n- `examples\u002Fweighted-ensemble-selection.json` — weighted ensemble example for `\u002Flav-ensemble-smoke`\n\nPrompt templates are also bundled for quick starts:\n\n- `\u002Fcompare-patches`\n- `\u002Faudit-candidate`\n- `\u002Fensemble-verifier`\n\n## Success criteria\n\nThis skill is successful when it:\n- chooses the strongest candidate rather than the most polished one,\n- makes the decision legible through explicit criteria,\n- incorporates deterministic evidence into the ranking,\n- surfaces uncertainty honestly,\n- produces a machine-readable artifact when needed,\n- helps the user rerun the comparison with sharper inputs when the result is weak.\n\n## Operating guidance\n\nStart with compare mode.\n\nUse **5 verifier samples** as the default for normal work. The Python runner\n(`scripts\u002Flav_runner.py`) defaults `n_verifications` to 5; the Pi extension\ndefaults `nVerifications` to `max(5, models.length)` for `pi-model-ensemble`\nand 5 for `gemini-python` \u002F `zai-coding-plan`. Explicit lower values\n(`n_verifications` in `1..8`) are still accepted for smoke tests, but treat\nanything below 5 as a high-variance decision aid, not a verdict.\n\n**Compare mode always runs A\u002FB and B\u002FA orderings.** Every pair \u002F criterion \u002F\nrepetition evaluates both `candidate_a → candidate_b` (original) and\n`candidate_b → candidate_a` (swapped). The canonical score for the swapped\nrun is fed back into the aggregate, so a position-only bias cannot pick a\nwinner on its own.\n\n**Low vote margin returns `tie` \u002F uncertain.** Pairwise compare mode counts\ncanonical per-repetition winners (ties count `0.5` for each candidate) and\ncomputes `vote_margin = max(votes) \u002F total_votes`. When `vote_margin \u003C 0.7`\nthe pair winner is forced to `tie` even if the mean scores differ — the\nnumeric scores stay visible, but the categorical decision is `uncertain`.\nAudit mode applies the same rule to its positive\u002Fnegative vote tally.\n\n**Candidate outputs are untrusted and scanned.** Every candidate answer is\nrouted through `scan_for_judge_manipulation`\n(`harness\u002Fsecurity\u002Fprompt_injection.py`) before rubric scoring. Flagged\ncandidates receive `judge-manipulation: \u003Cpattern>` weaknesses (penalizing\n`evidence_quality` and `safety_permission_fit`) and the run surfaces a\ntop-level warning in the format\n`candidate \u003Cid> contains judge-manipulation patterns: \u003Cflags>`. Legitimate\nsecurity discussion may be flagged and stays advisory plus penalty, not\nhard failure.\n\nIn `pi-model-ensemble`, if `nVerifications` is omitted, it defaults to\n`max(5, models.length)` so each model still gets at least one pass. The\ndefault panel is Kimi, MiniMax M3 from `minimax.io`, and GPT-5.5 via\nOpenAI\u002FCodex. Increase verifications only when the decision is high value\nand the candidate set is small.\n\nSupply shared evidence whenever the task has external ground truth.\n\nUse audit mode only as a fallback.\n\nTreat the verifier as a structured selector layered on top of deterministic evidence, not as a magical replacement for tests.\n",{"name":5,"description":7},"main","70395f201d2623f8048709ec3af66deb58ac4cfc","packages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier",0,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk","https:\u002F\u002Fomp.sh","https:\u002F\u002Fgithub.com\u002Fkingkillery","2026-11-14 17:53:41",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add kingkillery\u002Foh-my-pk --skill llm-as-verifier",{"tool":38,"command":39},"bunx","bunx skills add kingkillery\u002Foh-my-pk --skill llm-as-verifier",{"tool":41,"command":42},"pnpm","pnpm dlx skills add kingkillery\u002Foh-my-pk --skill llm-as-verifier","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002FSKILL.md",true,[46,52,56,60,64,68,73,76,80,84],{"path":47,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":50,"sourceHtmlUrl":51},"examples\u002Fcode-patch-selection.json","REFERENCE","参考文档",1908,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fexamples\u002Fcode-patch-selection.json",{"path":53,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":54,"sourceHtmlUrl":55},"examples\u002Fweighted-ensemble-selection.json",2394,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fexamples\u002Fweighted-ensemble-selection.json",{"path":57,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":58,"sourceHtmlUrl":59},"README.txt",795,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002FREADME.txt",{"path":61,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fcriteria-recipes.md",3294,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Freferences\u002Fcriteria-recipes.md",{"path":65,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fresearch-notes.md",6845,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Freferences\u002Fresearch-notes.md",{"path":69,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":27,"sourceHtmlUrl":72},"scripts\u002Fharness\u002F__init__.py","SCRIPT","脚本","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Fharness\u002F__init__.py",{"path":74,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":27,"sourceHtmlUrl":75},"scripts\u002Fharness\u002Ffusion\u002F__init__.py","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Fharness\u002Ffusion\u002F__init__.py",{"path":77,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":78,"sourceHtmlUrl":79},"scripts\u002Fharness\u002Ffusion\u002Fverifier_calibration.py",8184,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Fharness\u002Ffusion\u002Fverifier_calibration.py",{"path":81,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":82,"sourceHtmlUrl":83},"scripts\u002Fharness\u002Ffusion\u002Fverifier_scoring.py",1584,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Fharness\u002Ffusion\u002Fverifier_scoring.py",{"path":85,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":86,"sourceHtmlUrl":87},"scripts\u002Flav_runner.py",36036,"https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fllm-as-verifier\u002Fscripts\u002Flav_runner.py",[89,94,100,107,114,121],{"skillNo":90,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7572XYPFKX774ZDDEF4A210","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpython\u002Ffugu\u002F.agents\u002Fskills\u002Fllm-as-verifier\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M7572V10JTX07X4EV35RFEEW","pk-subagent-orchestrator","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fpk-subagent-orchestrator\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M7572QBZB4162KCDHBDEW4S3","pk-dynamic-workflows","This skill should be used when the user wants to spawn subagents with different models, configure heterogeneous candidate backends, implement dynamic model routing, or run multi-model fusion workflows. Covers mock, local, premium API (Anthropic\u002FOpenAI), budget API (Kimi\u002FMiniMax\u002FQwen), routing layers (9Router), and CLI backends.","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Fpk-dynamic-workflows\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7572FQHTHCGKWNA603A6209","fmh","Run fmh (fusion-meta-harness) — verifier fusion, evaluate-verifier, improve-verifier, promote, run-task. Orchestrates multi-agent candidate generation, rubric scoring, synthesis, and reliability gates for the pi-llm-as-verifier project.","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fverifier-extension\u002Fskills\u002Ffmh\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M7572EPWC4T7TCYHNTR7CEQT","greet","A greeting skill for testing","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fcoding-agent\u002Ftest\u002Fmarketplace\u002Ffixtures\u002Fvalid-marketplace\u002Fplugins\u002Fhello-plugin\u002Fskills\u002Fgreet\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":19},"01M7572D377961TGVQNQB7PFNH","valid-skill","A valid skill for testing purposes.","https:\u002F\u002Fgithub.com\u002Fkingkillery\u002Foh-my-pk\u002Fblob\u002Fmain\u002Fpackages\u002Fcoding-agent\u002Ftest\u002Ffixtures\u002Fskills\u002Fvalid-skill\u002FSKILL.md",[],[]]