[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6JGV517MFP7WD582E9ZFJ00":3,"skill-same:01M6JGV517MFP7WD582E9ZFJ00":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M6JGV517MFP7WD582E9ZFJ00","llm-output-validation",null,"Post-schema semantic validation checklist for structured LLM output. Use when writing code that calls an LLM and parses the response into a data model, reviewing code that consumes structured LLM output, or debugging cases where an LLM returned valid but wrong data.","muratovv\u002Fai-hats","muratovv","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F7502801?v=4","01M6JGCBN4KEJX012GPJVSH04T","Fedor Muratov","01M6JGCBSNMN7CW3RGTPPE0MN2",3,"MIT","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fusage\u002Fskills\u002Fllm-output-validation\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"llm output validation","validation","output","but","Wrong","for","calls","reviewing code","post","semantic","structured","writing","checklist","schema","use","model","data","code","debugging","valid","2026-11-10 10:12:30",false,"---\nname: llm-output-validation\ndescription: >-\n  Post-schema semantic validation checklist for structured LLM output. Use\n  when writing code that calls an LLM and parses the response into a data\n  model, reviewing code that consumes structured LLM output, or debugging\n  cases where an LLM returned valid but wrong data.\nlicense: MIT\n---\n# LLM Output Validation\n\nWhen an LLM produces structured data, schema validation (Pydantic, JSON Schema,\netc.) only guarantees **format correctness**. It cannot catch semantically wrong\nbut schema-valid output: hallucinated identifiers, out-of-scope selections,\nimpossible values. Treat structured LLM output the same way you treat untrusted\nuser input — validate **meaning**, not just shape.\n\n## When to Use\nThe boundary here is *after the bytes parse*: schema\u002FPydantic validation already\npassed, now validate **meaning** (hallucinated ids, out-of-scope picks,\nimpossible values). Not prompt design or model choice — for the Anthropic API\nsurface (params, tool-use, streaming) see the `claude-api` reference. Not generic\ninput validation either; this is specifically for output you got *from* an LLM.\n\n## Validation Checklist\n\nApply these checks **in code, immediately after `model_validate()` \u002F schema\nparse**. Raise on failure (loud, not a warning).\n\n### 1. Input Round-Trip\nEvery identifier in the output must exist in the input. No invented IDs.\n\n- If you passed `bundle_id=\"001\"`, the output must reference `\"001\"`, not `\"002\"`.\n- If you passed a list of filenames, every filename in the response must appear\n  in the original list.\n- Implementation: set\u002Fdict membership check on every ID field.\n\n### 2. Scope Subset\nWhen the output selects from the input (e.g., \"pick the top 3\"), verify\n`selection ⊆ input`.\n\n- If the input has items `[A, B, C]` and the output picks `[A, D]`, `D` is\n  hallucinated.\n- Implementation: `assert set(output.selected) \u003C= set(input.candidates)`.\n\n### 3. Value Bounds\nNumeric fields must be within the declared range.\n\n- Scores, ratings, confidence levels: check `0 \u003C= v \u003C= max`.\n- Counts: non-negative, bounded by input size.\n- Implementation: simple range assertions or Pydantic `Field(ge=0, le=10)`.\n\n### 4. Reference Integrity\nString references to entities (filenames, function names, URLs, paths) must\nresolve in the current context.\n\n- A cited filename must exist on disk (or in the provided listing).\n- A cited function must appear in the AST \u002F grep output.\n- Implementation: `Path(ref).exists()` or equivalent lookup.\n\n### 5. Semantic Plausibility\nCatch generic garbage the LLM emits when confused.\n\n- Non-empty fields where content is required.\n- No boilerplate filler (\"As an AI language model…\", \"I'm sorry…\").\n- Field lengths proportional to expected content (a one-word \"summary\" of a\n  1000-line file is suspicious).\n- Implementation: length checks, regex for known filler patterns.\n\n## Where to Put the Check\n\n```python\n# Immediately after schema parse — before ANY downstream use.\nresult = ResponseModel.model_validate(raw)\nvalidate_response_integrity(result, original_input)  # ← HERE\n# Now safe to use `result`.\n```\n\nRaise `ValueError` (or a domain-specific error) on failure. Do **not** log a\nwarning and continue — the whole point is to prevent downstream consumers from\nacting on garbage.\n\n## Case Study\n\nAn LLM in the (then-current) retrospective pipeline was asked to evaluate\n`bundle_id=\"001\"`. It returned a schema-valid result object with\n`bundle_id=\"002\"` — a hallucinated identifier. Pydantic didn't notice because\n`bundle_id` was typed as `str`, not constrained to the input. The fix: a\nthree-line integrity check after parse that verified `result.bundle_id ==\nexpected_bundle_id`. Trivial code, but it caught a class of bug that no amount\nof prompt engineering could reliably prevent.\n\n## Anti-Patterns\n- **Trusting Pydantic schema as a semantic guarantee** — schema validates shape,\n  not meaning.\n- **Logging a warning instead of raising** — downstream code will still consume\n  the bad data, just with a log line nobody reads.\n- **Deferring validation to a downstream consumer** — by the time the consumer\n  notices, the provenance is lost and the error is harder to debug.\n- **Adding constraints only to the prompt** — prompts are suggestions, not\n  contracts. Code is the only reliable enforcement.\n","# LLM Output Validation\n\nWhen an LLM produces structured data, schema validation (Pydantic, JSON Schema,\netc.) only guarantees **format correctness**. It cannot catch semantically wrong\nbut schema-valid output: hallucinated identifiers, out-of-scope selections,\nimpossible values. Treat structured LLM output the same way you treat untrusted\nuser input — validate **meaning**, not just shape.\n\n## When to Use\nThe boundary here is *after the bytes parse*: schema\u002FPydantic validation already\npassed, now validate **meaning** (hallucinated ids, out-of-scope picks,\nimpossible values). Not prompt design or model choice — for the Anthropic API\nsurface (params, tool-use, streaming) see the `claude-api` reference. Not generic\ninput validation either; this is specifically for output you got *from* an LLM.\n\n## Validation Checklist\n\nApply these checks **in code, immediately after `model_validate()` \u002F schema\nparse**. Raise on failure (loud, not a warning).\n\n### 1. Input Round-Trip\nEvery identifier in the output must exist in the input. No invented IDs.\n\n- If you passed `bundle_id=\"001\"`, the output must reference `\"001\"`, not `\"002\"`.\n- If you passed a list of filenames, every filename in the response must appear\n  in the original list.\n- Implementation: set\u002Fdict membership check on every ID field.\n\n### 2. Scope Subset\nWhen the output selects from the input (e.g., \"pick the top 3\"), verify\n`selection ⊆ input`.\n\n- If the input has items `[A, B, C]` and the output picks `[A, D]`, `D` is\n  hallucinated.\n- Implementation: `assert set(output.selected) \u003C= set(input.candidates)`.\n\n### 3. Value Bounds\nNumeric fields must be within the declared range.\n\n- Scores, ratings, confidence levels: check `0 \u003C= v \u003C= max`.\n- Counts: non-negative, bounded by input size.\n- Implementation: simple range assertions or Pydantic `Field(ge=0, le=10)`.\n\n### 4. Reference Integrity\nString references to entities (filenames, function names, URLs, paths) must\nresolve in the current context.\n\n- A cited filename must exist on disk (or in the provided listing).\n- A cited function must appear in the AST \u002F grep output.\n- Implementation: `Path(ref).exists()` or equivalent lookup.\n\n### 5. Semantic Plausibility\nCatch generic garbage the LLM emits when confused.\n\n- Non-empty fields where content is required.\n- No boilerplate filler (\"As an AI language model…\", \"I'm sorry…\").\n- Field lengths proportional to expected content (a one-word \"summary\" of a\n  1000-line file is suspicious).\n- Implementation: length checks, regex for known filler patterns.\n\n## Where to Put the Check\n\n```python\n# Immediately after schema parse — before ANY downstream use.\nresult = ResponseModel.model_validate(raw)\nvalidate_response_integrity(result, original_input)  # ← HERE\n# Now safe to use `result`.\n```\n\nRaise `ValueError` (or a domain-specific error) on failure. Do **not** log a\nwarning and continue — the whole point is to prevent downstream consumers from\nacting on garbage.\n\n## Case Study\n\nAn LLM in the (then-current) retrospective pipeline was asked to evaluate\n`bundle_id=\"001\"`. It returned a schema-valid result object with\n`bundle_id=\"002\"` — a hallucinated identifier. Pydantic didn't notice because\n`bundle_id` was typed as `str`, not constrained to the input. The fix: a\nthree-line integrity check after parse that verified `result.bundle_id ==\nexpected_bundle_id`. Trivial code, but it caught a class of bug that no amount\nof prompt engineering could reliably prevent.\n\n## Anti-Patterns\n- **Trusting Pydantic schema as a semantic guarantee** — schema validates shape,\n  not meaning.\n- **Logging a warning instead of raising** — downstream code will still consume\n  the bad data, just with a log line nobody reads.\n- **Deferring validation to a downstream consumer** — by the time the consumer\n  notices, the provenance is lost and the error is harder to debug.\n- **Adding constraints only to the prompt** — prompts are suggestions, not\n  contracts. Code is the only reliable enforcement.\n",{"name":5,"license":15,"description":7},"master","b4819add93efbaf4d80627118ec34680406b309b","packages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fusage\u002Fskills\u002Fllm-output-validation",1,"https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats","https:\u002F\u002Fgithub.com\u002Fmuratovv","2026-11-05 20:59:10",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add muratovv\u002Fai-hats --skill llm-output-validation",{"tool":57,"command":58},"bunx","bunx skills add muratovv\u002Fai-hats --skill llm-output-validation",{"tool":60,"command":61},"pnpm","pnpm dlx skills add muratovv\u002Fai-hats --skill llm-output-validation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fusage\u002Fskills\u002Fllm-output-validation\u002FSKILL.md",true,[],[66,91,115,137,161,175],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":39},"01M6PVFB1SM738R2SRSRSF7CY6","tool-evaluation-protocol","Time-bounded protocol for evaluating new tools, libraries, or methodologies before adoption. Use when evaluating a candidate dependency for the project, researching a methodology (e.g. ML\u002FLLM frameworks, testing approaches), or following up on a hype-driven recommendation.","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fexperimental\u002Fusage\u002Fskills\u002Ftool-evaluation-protocol\u002FSKILL.md",[],[73,74,75,76,77,78,79,80,81,82,33,83,84,85,86,87,88,24,89,90],"protocol","Evaluation","tool","tool evaluation protocol","testing","ml","libraries","adoption","methodology","Tools","dependency","up","frameworks","time","project","recommendation","bounded","candidate",{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":39},"01M6PVF7HATBEX1G7N50E2QT57","systematic-debugging","4-phase bug-fix protocol (evidence, pattern, hypothesis, verify). Use when investigating any error, failure, or unexpected behavior, handling bug reports from users or monitoring, or diagnosing test failures with non-obvious causes.","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fexperimental\u002Fusage\u002Fskills\u002Fsystematic-debugging\u002FSKILL.md",[],[98,37,99,100,73,101,102,103,104,105,33,106,107,108,109,110,111,112,113,114],"systematic debugging","systematic","test","monitoring","hypothesis","verify","behavior","fix","evidence","bug","reports","pattern","users","error","with","bug-fix","phase",{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":126,"publishedAt":39},"01M6PVF3MTQN3RPF73AN34856J","skill-template","Canonical template and validation guide for ai-hats skills. Use when creating a new skill, reviewing an existing one for structural compliance, or deciding which pattern fits a new behavior.","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fexperimental\u002Fusage\u002Fskills\u002Fskill-template\u002FSKILL.md",[121],{"categoryNo":122,"categoryName":123,"taxonomyType":124,"taxonomyTypeDesc":125},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[127,128,20,129,130,104,131,33,132,109,133,134,135,24,136],"template","skill template","compliance","hats","guide","new skill","structural","canonical","which","Existing",{"skillNo":138,"name":139,"nameZh":6,"description":140,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":141,"categories":142,"keywords":144,"publishedAt":39},"01M6PVF06SCV28QARM5MQPANC9","skill-optimization","Audit and refactor library components to eliminate redundancy and staleness. Use during periodic library maintenance, after adding multiple new skills or rules, when a role's token budget is too high, or after retrospectives identify component issues.","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fexperimental\u002Fusage\u002Fskills\u002Fskill-optimization\u002FSKILL.md",[143],{"categoryNo":122,"categoryName":123,"taxonomyType":124,"taxonomyTypeDesc":125},[145,146,147,148,149,150,151,152,153,154,33,155,156,157,158,159,160],"Optimization","skill optimization","components","component","issues","library","audit","refactor","Budget","Rules","periodic","maintenance","token","token budget","role","high",{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":167,"publishedAt":39},"01M6PVEXM9717016T9MW82HDCT","silent-fallback-gate","PreToolUse hook — refuses an Edit or Write that adds an inert broad except. Use when a Python edit was blocked for a silent fallback.","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fexperimental\u002Fusage\u002Fskills\u002Fsilent-fallback-gate\u002FSKILL.md",[],[168,169,170,171,172,173,33,174,24],"gate","fallback","silent fallback gate","python","write","edit","hook",{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":179,"categories":180,"keywords":182,"publishedAt":39},"01M6PVESM49C5NT7SB7N3F76VS","retro-to-framework","Convert an observation about agent behavior — from a retrospective or from a single incident — into a framework-level change (hook, rule, skill, skill update). Use after a retrospective identifies problems that are not project-specific, after an incident where an agent misapplied a component it had read, when CLAUDE.md band-aids accumulate (more than 3 per-project rules that could be generic), or when the same problem recurs across multiple projects.","https:\u002F\u002Fgithub.com\u002Fmuratovv\u002Fai-hats\u002Fblob\u002Fmaster\u002Fpackages\u002Fai-hats-library\u002Fsrc\u002Fai_hats_library\u002Fexperimental\u002Fusage\u002Fskills\u002Fretro-to-framework\u002FSKILL.md",[181],{"categoryNo":122,"categoryName":123,"taxonomyType":124,"taxonomyTypeDesc":125},[183,184,185,186,148,187,188,189,154,104,33,190,191,87,192,193,194,195,196,197],"framework","retro","retro to framework","change","retrospective","projects","convert","incident","across","problem","about","more","skill update","single","generic"]