[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M757NENAVBS0ADAERSRWB2TD":3,"skill-same:01M757NENAVBS0ADAERSRWB2TD":52},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":28,"commitSha":29,"skillDirectory":30,"forksCount":31,"repositoryHtmlUrl":32,"repositoryHomepageUrl":33,"ownerHtmlUrl":34,"ownerBlogUrl":35,"ownerDescription":36,"updatedAt":37,"install":38,"resources":51},"01M757NENAVBS0ADAERSRWB2TD","structured-output-and-tool-calling",null,"Reference-grade guide to getting reliable structured output and tool\u002Ffunction calls out of LLMs — JSON mode vs constrained\u002Fgrammar-guided decoding, response_format json_schema and tool-schema coercion, Zod\u002FPydantic validation at the boundary, the validate→repair→escalate→fallback loop, tool-argument validation, idempotency for safe retries, fallback chains, and evals. Use when an LLM must return parseable JSON, fill typed tool arguments, or drive deterministic downstream code.","jpoindexter\u002FVanta","jpoindexter","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F2729617?v=4","01M72VRRYY1CP7RPF3B7GHH0CR","01M757GWDBT6RD147ZZ79N4MEX",4,"MIT","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fai-engineering-skills\u002Fstructured-output-and-tool-calling\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: structured-output-and-tool-calling\ndescription: Reference-grade guide to getting reliable structured output and tool\u002Ffunction calls out of LLMs — JSON mode vs constrained\u002Fgrammar-guided decoding, response_format json_schema and tool-schema coercion, Zod\u002FPydantic validation at the boundary, the validate→repair→escalate→fallback loop, tool-argument validation, idempotency for safe retries, fallback chains, and evals. Use when an LLM must return parseable JSON, fill typed tool arguments, or drive deterministic downstream code.\ntags: [ai-engineering, structured-output, function-calling, json-schema]\n---\n\n# Structured Output & Tool Calling Reliability\n\nThe moment an LLM's output feeds deterministic code — a parser, a database write, a payment call — \"usually valid\" is a production incident waiting on a percentage. This skill is about closing that gap: making the *structure* guaranteed, the *contents* validated, the *execution* idempotent, and the *failures* recoverable.\n\nCore mental model — three independent guarantees, do not conflate them:\n\n1. **Structural validity** (is it parseable JSON matching the schema shape?) — solved by *constrained decoding*.\n2. **Semantic validity** (are the values correct, in-range, referencing real things?) — solved only by *validation + repair*.\n3. **Execution safety** (does running it twice double-charge?) — solved only by *idempotency*.\n\nConstrained decoding gives you #1 for free and *nothing else*. You still own #2 and #3.\n\n---\n\n## 1. Structured output: JSON mode vs constrained decoding\n\nThe single most important distinction in this whole skill:\n\n| Mechanism | What it does | Guarantee | Still need to validate? |\n|---|---|---|---|\n| **JSON mode** (`response_format: {type:\"json_object\"}`) | Biases the model toward JSON via prompt + light logit nudging | Best-effort. Can still truncate, emit wrong types, omit fields | **Yes — always** |\n| **Constrained \u002F grammar-guided decoding** (`json_schema` strict, GBNF, Outlines) | At every sampling step, *masks out* tokens that would violate the grammar\u002Fschema | Structurally valid by construction — cannot emit a token that breaks the schema | Yes, for *semantics* — but never for parse\u002Fshape |\n| **Free-text + parse** (no mode) | Nothing; you `JSON.parse` whatever comes out | None. Prose wrappers, markdown fences, commentary | Yes, and brace for failure |\n\n**Why constrained decoding eliminates malformed JSON:** the decoder maintains a grammar state machine. Before sampling each token, it computes the set of tokens that keep the partial output grammar-valid and sets every other token's logit to `-inf`. The model literally cannot sample a `}` where the schema requires more fields, or a string where it requires an integer. There is no \"the model usually closes its braces\" — closing is the only legal move. This is the difference between *guaranteed-valid* and *best-effort*. If you have access to constrained decoding for a schema-shaped task, use it; it deletes an entire class of bugs.\n\n### Provider mapping (capability-level — verify exact model support before relying on it)\n\n- **OpenAI Structured Outputs** — `response_format: {type:\"json_schema\", json_schema:{name, strict:true, schema:{...}}}`. With `strict:true` this is *constrained decoding*: output is guaranteed to match the JSON Schema (subset: every field `required`, `additionalProperties:false`, limited type expressivity). The older `{type:\"json_object\"}` is JSON *mode* — best-effort, validate it.\n- **OpenAI tool calls** — pass `tools` with JSON-Schema `parameters`; `strict:true` on a function applies the same constrained guarantee to its arguments.\n- **Anthropic (Claude)** — no native `json_schema` response_format. The stable pattern is **tool-schema coercion**: define a single tool whose `input_schema` is your target schema, then force it with `tool_choice:{type:\"tool\", name:\"...\"}`. The model fills the tool's arguments = your structured object. Read it from the `tool_use` block. Validation still required; coercion biases shape but is not token-masked grammar enforcement.\n- **Open \u002F local** — `Outlines` (regex + JSON-Schema → FSM-masked logits), `llama.cpp` **GBNF** grammars, `vLLM`\u002F`SGLang`\u002F`TGI` guided-decoding backends (`guided_json`, `guided_grammar`). These are *true* constrained decoding and work with any JSON Schema or hand-written grammar.\n\n```text\nGBNF sketch — a grammar IS the contract:\nroot   ::= \"{\" ws \"\\\"action\\\"\" ws \":\" ws action ws \",\" ws \"\\\"id\\\"\" ws \":\" ws integer ws \"}\"\naction ::= \"\\\"create\\\"\" | \"\\\"update\\\"\" | \"\\\"delete\\\"\"   # enum, structurally enforced\ninteger::= \"-\"? [0-9]+\n```\n\n### Validate at the boundary — always, even with constrained decoding\n\nConstrained decoding guarantees *shape*, not *meaning*. A schema-valid object can still say `quantity: -5` or `country: \"Wakanda\"`. Put a real validator at the trust boundary and parse into a typed object so the rest of your code touches only validated data.\n\n```ts\n\u002F\u002F Zod — single source of truth: feed .toJSON()→JSON Schema to the model, parse the reply\nconst Order = z.object({\n  action: z.enum([\"create\", \"update\", \"delete\"]),\n  quantity: z.number().int().positive(),        \u002F\u002F catches -5 that the schema \"shape\" allows\n  sku: z.string().regex(\u002F^[A-Z]{3}-\\d{4}$\u002F),\n});\nconst parsed = Order.safeParse(JSON.parse(raw));\nif (!parsed.success) return repairLoop(raw, parsed.error);   \u002F\u002F see §3\n```\n\n```python\n# Pydantic — model_validate_json does parse + validate + coerce in one step\nclass Order(BaseModel):\n    action: Literal[\"create\", \"update\", \"delete\"]\n    quantity: PositiveInt\n    sku: constr(pattern=r\"^[A-Z]{3}-\\d{4}$\")\ntry:\n    order = Order.model_validate_json(raw)\nexcept ValidationError as e:\n    return repair_loop(raw, e)\n```\n\n**Do:** derive the model-facing JSON Schema *from* your Zod\u002FPydantic type (one source of truth). **Don't:** hand-write a JSON Schema for the API and a separate validator — they drift, and the gap is exactly where bad data flows.\n\n---\n\n## 2. Failure modes of structured output\n\nEven good setups fail. Know the catalogue so you can detect and repair each one specifically.\n\n| Failure mode | Cause | Detection | Fix |\n|---|---|---|---|\n| **Malformed JSON** | No constrained decoding; weak model | `JSON.parse` throws | Constrained decoding; else repair loop |\n| **Truncation** | `max_tokens` hit mid-object; stream cut | Parse error at EOF; `finish_reason:\"length\"` | Raise `max_tokens`; partial-parse; continue\u002Frepair |\n| **Wrong types** | `\"5\"` vs `5`, `\"true\"` vs `true` | Validator type error | Zod\u002FPydantic coercion; strict schema |\n| **Extra prose** | \"Here is the JSON: ```json …```\" | Leading\u002Ftrailing non-JSON | JSON mode; strip fences; extract first `{…}` |\n| **Hallucinated fields** | Model invents keys | `additionalProperties:false` \u002F Zod `.strict()` | Reject + repair |\n| **Missing required fields** | Model omits | Validator \"required\" error | `strict:true`; repair with the missing field named |\n| **Enum violation** | Value outside allowed set | Validator enum error | Constrained decoding enforces enums; else repair |\n| **Nested-depth \u002F array errors** | Deep or recursive schemas confuse model | Validator path error | Flatten schema; fewer levels; provide an example |\n| **Empty \u002F null where required** | Model unsure, emits `null` | Validator non-nullable error | Make optionality explicit; ask for a sentinel |\n\n### The repair loop\n\nTreat the validator's error as a *signal to the model*, not just a 500 to the user. Bounded, escalating, idempotent.\n\n```text\nvalidate(output)\n  ├─ ok ────────────────────────────────► return typed object\n  └─ fail\n       ├─ attempts \u003C N (e.g. 2):\n       │     feed back: original output + the EXACT validator error\n       │     (\"field `quantity` must be a positive integer, got -5; return corrected JSON only\")\n       │     re-call SAME model → validate again (loop)\n       ├─ attempts exhausted:\n       │     ESCALATE → stronger model, same prompt + error\n       └─ still failing:\n             FALLBACK → deterministic default \u002F queue for human \u002F typed error to caller\n```\n\nRules that make repair work:\n\n- **Bound it.** 1–2 repair attempts, then escalate. Unbounded retry loops burn tokens and latency on a model that's stuck.\n- **Feed the real error back verbatim.** \"Invalid input\" teaches the model nothing. \"`sku` must match `^[A-Z]{3}-\\d{4}$`, got `abc`\" gets a fix in one shot.\n- **Idempotent repair.** A repair retry must not have side effects — never execute the action mid-repair; only re-derive the object. Execution happens once, *after* a clean validate.\n- **Partial-parse \u002F streaming JSON.** When streaming, use a tolerant incremental parser (`partial-json`, `best-effort-json-parser`, `jsonrepair`) to render in-progress UI, but **only validate and act on the final completed object.** Never trigger a tool from a partial parse.\n- **Truncation is special.** If `finish_reason === \"length\"`, the object isn't wrong — it's *incomplete*. Don't \"repair\" it as malformed; re-request with higher `max_tokens` or use a continuation strategy.\n\n---\n\n## 3. Function \u002F tool calling reliability\n\nA tool call is structured output whose schema is a function signature — every structured-output rule applies, plus the brutal new fact: **the output triggers real-world side effects.** A hallucinated number in a summary is cosmetic; a hallucinated argument to `refund(amount)` is money.\n\n### Tool contracts — design for the model, not just the compiler\n\n```ts\n{\n  name: \"search_orders\",                       \u002F\u002F verb_noun, unambiguous, no overlap with siblings\n  description: \"Search orders by customer email. Returns up to `limit` orders, newest first. \" +\n               \"Use ONLY for lookups — does not modify anything.\",   \u002F\u002F says what it does AND when to use it\n  parameters: {\n    type: \"object\",\n    properties: {\n      email:  { type: \"string\", format: \"email\", description: \"Exact customer email\" },\n      status: { type: \"string\", enum: [\"open\",\"shipped\",\"cancelled\"] },  \u002F\u002F constrain, don't free-text\n      limit:  { type: \"integer\", minimum: 1, maximum: 50, default: 20 },\n    },\n    required: [\"email\"],                        \u002F\u002F required vs optional, explicit\n    additionalProperties: false,\n  },\n}\n```\n\n| Do | Don't |\n|---|---|\n| One clear responsibility per tool, verb-first name | `handle_data`, `do_action`, vague catch-alls |\n| Description states *what* and *when to use \u002F not use* | Description restates the name |\n| `enum` every closed set; `min\u002Fmax`\u002F`format` every value | Free-text params the model can hallucinate |\n| Mark `required` vs optional; give sane `default`s | Everything optional → model omits the field you need |\n| Include 1 example call in the system prompt for tricky tools | Assume the model infers usage from the name |\n| Cap the toolset (see degradation below) | 40 tools in one call and hope |\n\n### Validate arguments BEFORE executing — never trust LLM args\n\nThe model's tool call is a *request*, not an authorization. Re-validate against the schema **and** business rules in your code before the side effect runs.\n\n```ts\nfunction dispatch(call: ToolCall) {\n  const tool = registry[call.name];\n  if (!tool) return toolError(call, `Unknown tool '${call.name}'. Available: ${Object.keys(registry)}`);  \u002F\u002F hallucinated tool\n  const args = tool.schema.safeParse(call.arguments);\n  if (!args.success) return toolError(call, args.error.message);   \u002F\u002F hallucinated\u002Finvalid args → back to model\n  if (!authorize(tool, args.data)) return toolError(call, \"Not permitted\");  \u002F\u002F business rule, not the LLM's call\n  return tool.run(args.data);   \u002F\u002F only now does anything happen\n}\n```\n\n**The hallucinated tool call** comes in two flavors, both caught above by *not trusting the call*:\n- **Tool that doesn't exist** — model invents `delete_everything`. Catch: registry lookup fails → return a tool-result error listing real tools; the model self-corrects next turn.\n- **Wrong \u002F invented arguments** — right tool, garbage args (made-up order ID, out-of-enum status). Catch: schema validation fails → return the error as the tool result; do **not** execute.\n\nReturning the error *as a tool result* (not throwing) keeps the model in the loop and lets it retry with corrected input — far better than crashing the run.\n\n### Tool-choice forcing & parallel calls\n\n- **`tool_choice`** — `auto` (model decides), `required`\u002F`any` (must call *some* tool), or force a *specific* tool (= structured-output coercion, §1). Force when you know a tool is needed; leave `auto` for genuine routing.\n- **Parallel tool calls** — providers may return several `tool_use` blocks in one turn. Execute them, but: validate each independently, watch for the same write issued twice (dedupe via idempotency, §4), and return *all* results before the next model turn or you desync the conversation.\n\n### Too-many-tools degradation — cap and route\n\nAccuracy of tool selection drops as the toolset grows (more near-synonym descriptions to disambiguate, more schema in context). Past ~15–20 tools, expect wrong-tool picks and ignored tools.\n\n**Mitigations, cheapest first:** (1) **cap** the tools exposed per call to the handful relevant to the task; (2) **route** — a cheap first-stage model\u002Fretriever picks the relevant 5–10 tools, then the main call sees only those; (3) **namespace\u002Fgroup** tools and expose one group at a time; (4) **merge** near-duplicate tools behind one with an `action` enum. Don't paste your entire API surface into every request.\n\n---\n\n## 4. Idempotency — so retries don't double-charge\n\nRepair loops, parallel calls, network retries, and at-least-once queues all mean a write tool **will** be invoked more than once for one logical intent. If executing it twice does damage, you have a latent incident. Design every write tool so duplicate execution is a no-op.\n\n| Tool kind | Retry safety | Rule |\n|---|---|---|\n| **Read** (`search`, `get`, `list`) | Inherently safe | Retry freely; no key needed |\n| **Idempotent write** (`set_status`, `upsert`) | Safe by design | Same input → same end state |\n| **Non-idempotent write** (`charge`, `send_email`, `append`) | **Dangerous** | Require an idempotency key; dedupe |\n| **Destructive** (`delete`, `drop`, `payout`) | Dangerous + irreversible | Idempotency key **+** explicit human confirmation |\n\nPatterns:\n\n- **Idempotency keys.** Derive a stable key from the logical intent (`hash(tool + canonical_args)` or a request-scoped UUID minted once before any retry). Pass it to the downstream system (Stripe-style `Idempotency-Key`) or store `seen_keys`; on a duplicate key, return the cached prior result instead of re-executing.\n- **Dedupe at the boundary.** Before executing a write, check whether this exact `(key)` already ran this session. The repair loop must reuse the *same* key across attempts — minting a new key per retry defeats the whole mechanism.\n- **Read vs write safety.** Auto-execute read tools; gate write tools through validation + idempotency; gate destructive tools through an explicit confirmation step (surface the action to a human\u002Fapproval queue before running).\n- **Make the model's retry cheap and safe.** Because tool-arg errors are fed back for repair, the *same* call can arrive twice with one tweak — idempotency keys keyed on intent (not on the exact bytes) absorb that.\n\n---\n\n## 5. Fallback chains\n\nCompose the mechanisms into a single degrade-gracefully path. Each stage is strictly more expensive\u002Fslower than the last; stop at the first success.\n\n```text\n1. Constrained decoding (json_schema strict \u002F GBNF)   ← structurally valid, cheapest\n2. Validate (Zod\u002FPydantic)                            ← semantic check\n3. Repair loop ×N (feed error back, same model)       ← fixes most semantic misses\n4. Escalate to a stronger model (same prompt+error)   ← for genuinely hard cases\n5. Deterministic default \u002F human review \u002F typed error ← never crash the caller\n```\n\nThe non-negotiable bottom rung: **a typed, actionable failure, never an unhandled throw or a silent empty object.** Downstream code should receive `Result\u003CT, StructuredOutputError>`, not a surprise.\n\n---\n\n## 6. Evals for structured output\n\nYou cannot improve reliability you don't measure. Run these continuously on a fixed input set across model\u002Fprompt\u002Fschema changes.\n\n| Metric | Definition | Target |\n|---|---|---|\n| **Valid-rate** | % outputs that parse + pass schema on first try | The headline number; track per model |\n| **Repair-rate** | % needing ≥1 repair attempt | High → fix schema\u002Fprompt, not just retry |\n| **Repair-success** | % of repairs that converge within N | Low → escalate sooner |\n| **Field accuracy** | per-field correctness vs gold labels | Catches enum\u002Ftype drift a valid-rate hides |\n| **Tool-selection accuracy** | right tool chosen for the intent | Drops as toolset grows → triggers routing |\n| **Arg accuracy** | tool args correct vs expected | Separate from selecting the right tool |\n| **Hallucinated-call rate** | calls to nonexistent tools \u002F invalid args | Should trend to ~0 with good contracts |\n\n**Do:** keep a golden set of inputs with expected structured outputs; assert per-field, not just \"did it parse.\" A 100% valid-rate with 60% field accuracy is a worse failure than a parse error — it *looks* fine and ships wrong data. **Don't:** measure only valid-rate; constrained decoding can make it 100% while the *contents* are nonsense.\n\n---\n\n## Quick reference — decision order\n\n1. Can I use **constrained decoding** for this schema? → Do it. Malformed JSON gone.\n2. **Validate** the result at the boundary with Zod\u002FPydantic into a typed object — always, even after #1.\n3. On failure: **repair** (≤2×, feed the exact error) → **escalate** model → **fallback** to a typed default\u002Fhuman.\n4. For tools: tight **contracts**, **validate args before executing**, catch **hallucinated** calls by not trusting them, **cap\u002Froute** when tools are many.\n5. Make every write **idempotent**; gate destructive actions on confirmation.\n6. **Eval** valid-rate *and* field accuracy continuously — the second is the one that lies quietly.\n","# Structured Output & Tool Calling Reliability\n\nThe moment an LLM's output feeds deterministic code — a parser, a database write, a payment call — \"usually valid\" is a production incident waiting on a percentage. This skill is about closing that gap: making the *structure* guaranteed, the *contents* validated, the *execution* idempotent, and the *failures* recoverable.\n\nCore mental model — three independent guarantees, do not conflate them:\n\n1. **Structural validity** (is it parseable JSON matching the schema shape?) — solved by *constrained decoding*.\n2. **Semantic validity** (are the values correct, in-range, referencing real things?) — solved only by *validation + repair*.\n3. **Execution safety** (does running it twice double-charge?) — solved only by *idempotency*.\n\nConstrained decoding gives you #1 for free and *nothing else*. You still own #2 and #3.\n\n---\n\n## 1. Structured output: JSON mode vs constrained decoding\n\nThe single most important distinction in this whole skill:\n\n| Mechanism | What it does | Guarantee | Still need to validate? |\n|---|---|---|---|\n| **JSON mode** (`response_format: {type:\"json_object\"}`) | Biases the model toward JSON via prompt + light logit nudging | Best-effort. Can still truncate, emit wrong types, omit fields | **Yes — always** |\n| **Constrained \u002F grammar-guided decoding** (`json_schema` strict, GBNF, Outlines) | At every sampling step, *masks out* tokens that would violate the grammar\u002Fschema | Structurally valid by construction — cannot emit a token that breaks the schema | Yes, for *semantics* — but never for parse\u002Fshape |\n| **Free-text + parse** (no mode) | Nothing; you `JSON.parse` whatever comes out | None. Prose wrappers, markdown fences, commentary | Yes, and brace for failure |\n\n**Why constrained decoding eliminates malformed JSON:** the decoder maintains a grammar state machine. Before sampling each token, it computes the set of tokens that keep the partial output grammar-valid and sets every other token's logit to `-inf`. The model literally cannot sample a `}` where the schema requires more fields, or a string where it requires an integer. There is no \"the model usually closes its braces\" — closing is the only legal move. This is the difference between *guaranteed-valid* and *best-effort*. If you have access to constrained decoding for a schema-shaped task, use it; it deletes an entire class of bugs.\n\n### Provider mapping (capability-level — verify exact model support before relying on it)\n\n- **OpenAI Structured Outputs** — `response_format: {type:\"json_schema\", json_schema:{name, strict:true, schema:{...}}}`. With `strict:true` this is *constrained decoding*: output is guaranteed to match the JSON Schema (subset: every field `required`, `additionalProperties:false`, limited type expressivity). The older `{type:\"json_object\"}` is JSON *mode* — best-effort, validate it.\n- **OpenAI tool calls** — pass `tools` with JSON-Schema `parameters`; `strict:true` on a function applies the same constrained guarantee to its arguments.\n- **Anthropic (Claude)** — no native `json_schema` response_format. The stable pattern is **tool-schema coercion**: define a single tool whose `input_schema` is your target schema, then force it with `tool_choice:{type:\"tool\", name:\"...\"}`. The model fills the tool's arguments = your structured object. Read it from the `tool_use` block. Validation still required; coercion biases shape but is not token-masked grammar enforcement.\n- **Open \u002F local** — `Outlines` (regex + JSON-Schema → FSM-masked logits), `llama.cpp` **GBNF** grammars, `vLLM`\u002F`SGLang`\u002F`TGI` guided-decoding backends (`guided_json`, `guided_grammar`). These are *true* constrained decoding and work with any JSON Schema or hand-written grammar.\n\n```text\nGBNF sketch — a grammar IS the contract:\nroot   ::= \"{\" ws \"\\\"action\\\"\" ws \":\" ws action ws \",\" ws \"\\\"id\\\"\" ws \":\" ws integer ws \"}\"\naction ::= \"\\\"create\\\"\" | \"\\\"update\\\"\" | \"\\\"delete\\\"\"   # enum, structurally enforced\ninteger::= \"-\"? [0-9]+\n```\n\n### Validate at the boundary — always, even with constrained decoding\n\nConstrained decoding guarantees *shape*, not *meaning*. A schema-valid object can still say `quantity: -5` or `country: \"Wakanda\"`. Put a real validator at the trust boundary and parse into a typed object so the rest of your code touches only validated data.\n\n```ts\n\u002F\u002F Zod — single source of truth: feed .toJSON()→JSON Schema to the model, parse the reply\nconst Order = z.object({\n  action: z.enum([\"create\", \"update\", \"delete\"]),\n  quantity: z.number().int().positive(),        \u002F\u002F catches -5 that the schema \"shape\" allows\n  sku: z.string().regex(\u002F^[A-Z]{3}-\\d{4}$\u002F),\n});\nconst parsed = Order.safeParse(JSON.parse(raw));\nif (!parsed.success) return repairLoop(raw, parsed.error);   \u002F\u002F see §3\n```\n\n```python\n# Pydantic — model_validate_json does parse + validate + coerce in one step\nclass Order(BaseModel):\n    action: Literal[\"create\", \"update\", \"delete\"]\n    quantity: PositiveInt\n    sku: constr(pattern=r\"^[A-Z]{3}-\\d{4}$\")\ntry:\n    order = Order.model_validate_json(raw)\nexcept ValidationError as e:\n    return repair_loop(raw, e)\n```\n\n**Do:** derive the model-facing JSON Schema *from* your Zod\u002FPydantic type (one source of truth). **Don't:** hand-write a JSON Schema for the API and a separate validator — they drift, and the gap is exactly where bad data flows.\n\n---\n\n## 2. Failure modes of structured output\n\nEven good setups fail. Know the catalogue so you can detect and repair each one specifically.\n\n| Failure mode | Cause | Detection | Fix |\n|---|---|---|---|\n| **Malformed JSON** | No constrained decoding; weak model | `JSON.parse` throws | Constrained decoding; else repair loop |\n| **Truncation** | `max_tokens` hit mid-object; stream cut | Parse error at EOF; `finish_reason:\"length\"` | Raise `max_tokens`; partial-parse; continue\u002Frepair |\n| **Wrong types** | `\"5\"` vs `5`, `\"true\"` vs `true` | Validator type error | Zod\u002FPydantic coercion; strict schema |\n| **Extra prose** | \"Here is the JSON: ```json …```\" | Leading\u002Ftrailing non-JSON | JSON mode; strip fences; extract first `{…}` |\n| **Hallucinated fields** | Model invents keys | `additionalProperties:false` \u002F Zod `.strict()` | Reject + repair |\n| **Missing required fields** | Model omits | Validator \"required\" error | `strict:true`; repair with the missing field named |\n| **Enum violation** | Value outside allowed set | Validator enum error | Constrained decoding enforces enums; else repair |\n| **Nested-depth \u002F array errors** | Deep or recursive schemas confuse model | Validator path error | Flatten schema; fewer levels; provide an example |\n| **Empty \u002F null where required** | Model unsure, emits `null` | Validator non-nullable error | Make optionality explicit; ask for a sentinel |\n\n### The repair loop\n\nTreat the validator's error as a *signal to the model*, not just a 500 to the user. Bounded, escalating, idempotent.\n\n```text\nvalidate(output)\n  ├─ ok ────────────────────────────────► return typed object\n  └─ fail\n       ├─ attempts \u003C N (e.g. 2):\n       │     feed back: original output + the EXACT validator error\n       │     (\"field `quantity` must be a positive integer, got -5; return corrected JSON only\")\n       │     re-call SAME model → validate again (loop)\n       ├─ attempts exhausted:\n       │     ESCALATE → stronger model, same prompt + error\n       └─ still failing:\n             FALLBACK → deterministic default \u002F queue for human \u002F typed error to caller\n```\n\nRules that make repair work:\n\n- **Bound it.** 1–2 repair attempts, then escalate. Unbounded retry loops burn tokens and latency on a model that's stuck.\n- **Feed the real error back verbatim.** \"Invalid input\" teaches the model nothing. \"`sku` must match `^[A-Z]{3}-\\d{4}$`, got `abc`\" gets a fix in one shot.\n- **Idempotent repair.** A repair retry must not have side effects — never execute the action mid-repair; only re-derive the object. Execution happens once, *after* a clean validate.\n- **Partial-parse \u002F streaming JSON.** When streaming, use a tolerant incremental parser (`partial-json`, `best-effort-json-parser`, `jsonrepair`) to render in-progress UI, but **only validate and act on the final completed object.** Never trigger a tool from a partial parse.\n- **Truncation is special.** If `finish_reason === \"length\"`, the object isn't wrong — it's *incomplete*. Don't \"repair\" it as malformed; re-request with higher `max_tokens` or use a continuation strategy.\n\n---\n\n## 3. Function \u002F tool calling reliability\n\nA tool call is structured output whose schema is a function signature — every structured-output rule applies, plus the brutal new fact: **the output triggers real-world side effects.** A hallucinated number in a summary is cosmetic; a hallucinated argument to `refund(amount)` is money.\n\n### Tool contracts — design for the model, not just the compiler\n\n```ts\n{\n  name: \"search_orders\",                       \u002F\u002F verb_noun, unambiguous, no overlap with siblings\n  description: \"Search orders by customer email. Returns up to `limit` orders, newest first. \" +\n               \"Use ONLY for lookups — does not modify anything.\",   \u002F\u002F says what it does AND when to use it\n  parameters: {\n    type: \"object\",\n    properties: {\n      email:  { type: \"string\", format: \"email\", description: \"Exact customer email\" },\n      status: { type: \"string\", enum: [\"open\",\"shipped\",\"cancelled\"] },  \u002F\u002F constrain, don't free-text\n      limit:  { type: \"integer\", minimum: 1, maximum: 50, default: 20 },\n    },\n    required: [\"email\"],                        \u002F\u002F required vs optional, explicit\n    additionalProperties: false,\n  },\n}\n```\n\n| Do | Don't |\n|---|---|\n| One clear responsibility per tool, verb-first name | `handle_data`, `do_action`, vague catch-alls |\n| Description states *what* and *when to use \u002F not use* | Description restates the name |\n| `enum` every closed set; `min\u002Fmax`\u002F`format` every value | Free-text params the model can hallucinate |\n| Mark `required` vs optional; give sane `default`s | Everything optional → model omits the field you need |\n| Include 1 example call in the system prompt for tricky tools | Assume the model infers usage from the name |\n| Cap the toolset (see degradation below) | 40 tools in one call and hope |\n\n### Validate arguments BEFORE executing — never trust LLM args\n\nThe model's tool call is a *request*, not an authorization. Re-validate against the schema **and** business rules in your code before the side effect runs.\n\n```ts\nfunction dispatch(call: ToolCall) {\n  const tool = registry[call.name];\n  if (!tool) return toolError(call, `Unknown tool '${call.name}'. Available: ${Object.keys(registry)}`);  \u002F\u002F hallucinated tool\n  const args = tool.schema.safeParse(call.arguments);\n  if (!args.success) return toolError(call, args.error.message);   \u002F\u002F hallucinated\u002Finvalid args → back to model\n  if (!authorize(tool, args.data)) return toolError(call, \"Not permitted\");  \u002F\u002F business rule, not the LLM's call\n  return tool.run(args.data);   \u002F\u002F only now does anything happen\n}\n```\n\n**The hallucinated tool call** comes in two flavors, both caught above by *not trusting the call*:\n- **Tool that doesn't exist** — model invents `delete_everything`. Catch: registry lookup fails → return a tool-result error listing real tools; the model self-corrects next turn.\n- **Wrong \u002F invented arguments** — right tool, garbage args (made-up order ID, out-of-enum status). Catch: schema validation fails → return the error as the tool result; do **not** execute.\n\nReturning the error *as a tool result* (not throwing) keeps the model in the loop and lets it retry with corrected input — far better than crashing the run.\n\n### Tool-choice forcing & parallel calls\n\n- **`tool_choice`** — `auto` (model decides), `required`\u002F`any` (must call *some* tool), or force a *specific* tool (= structured-output coercion, §1). Force when you know a tool is needed; leave `auto` for genuine routing.\n- **Parallel tool calls** — providers may return several `tool_use` blocks in one turn. Execute them, but: validate each independently, watch for the same write issued twice (dedupe via idempotency, §4), and return *all* results before the next model turn or you desync the conversation.\n\n### Too-many-tools degradation — cap and route\n\nAccuracy of tool selection drops as the toolset grows (more near-synonym descriptions to disambiguate, more schema in context). Past ~15–20 tools, expect wrong-tool picks and ignored tools.\n\n**Mitigations, cheapest first:** (1) **cap** the tools exposed per call to the handful relevant to the task; (2) **route** — a cheap first-stage model\u002Fretriever picks the relevant 5–10 tools, then the main call sees only those; (3) **namespace\u002Fgroup** tools and expose one group at a time; (4) **merge** near-duplicate tools behind one with an `action` enum. Don't paste your entire API surface into every request.\n\n---\n\n## 4. Idempotency — so retries don't double-charge\n\nRepair loops, parallel calls, network retries, and at-least-once queues all mean a write tool **will** be invoked more than once for one logical intent. If executing it twice does damage, you have a latent incident. Design every write tool so duplicate execution is a no-op.\n\n| Tool kind | Retry safety | Rule |\n|---|---|---|\n| **Read** (`search`, `get`, `list`) | Inherently safe | Retry freely; no key needed |\n| **Idempotent write** (`set_status`, `upsert`) | Safe by design | Same input → same end state |\n| **Non-idempotent write** (`charge`, `send_email`, `append`) | **Dangerous** | Require an idempotency key; dedupe |\n| **Destructive** (`delete`, `drop`, `payout`) | Dangerous + irreversible | Idempotency key **+** explicit human confirmation |\n\nPatterns:\n\n- **Idempotency keys.** Derive a stable key from the logical intent (`hash(tool + canonical_args)` or a request-scoped UUID minted once before any retry). Pass it to the downstream system (Stripe-style `Idempotency-Key`) or store `seen_keys`; on a duplicate key, return the cached prior result instead of re-executing.\n- **Dedupe at the boundary.** Before executing a write, check whether this exact `(key)` already ran this session. The repair loop must reuse the *same* key across attempts — minting a new key per retry defeats the whole mechanism.\n- **Read vs write safety.** Auto-execute read tools; gate write tools through validation + idempotency; gate destructive tools through an explicit confirmation step (surface the action to a human\u002Fapproval queue before running).\n- **Make the model's retry cheap and safe.** Because tool-arg errors are fed back for repair, the *same* call can arrive twice with one tweak — idempotency keys keyed on intent (not on the exact bytes) absorb that.\n\n---\n\n## 5. Fallback chains\n\nCompose the mechanisms into a single degrade-gracefully path. Each stage is strictly more expensive\u002Fslower than the last; stop at the first success.\n\n```text\n1. Constrained decoding (json_schema strict \u002F GBNF)   ← structurally valid, cheapest\n2. Validate (Zod\u002FPydantic)                            ← semantic check\n3. Repair loop ×N (feed error back, same model)       ← fixes most semantic misses\n4. Escalate to a stronger model (same prompt+error)   ← for genuinely hard cases\n5. Deterministic default \u002F human review \u002F typed error ← never crash the caller\n```\n\nThe non-negotiable bottom rung: **a typed, actionable failure, never an unhandled throw or a silent empty object.** Downstream code should receive `Result\u003CT, StructuredOutputError>`, not a surprise.\n\n---\n\n## 6. Evals for structured output\n\nYou cannot improve reliability you don't measure. Run these continuously on a fixed input set across model\u002Fprompt\u002Fschema changes.\n\n| Metric | Definition | Target |\n|---|---|---|\n| **Valid-rate** | % outputs that parse + pass schema on first try | The headline number; track per model |\n| **Repair-rate** | % needing ≥1 repair attempt | High → fix schema\u002Fprompt, not just retry |\n| **Repair-success** | % of repairs that converge within N | Low → escalate sooner |\n| **Field accuracy** | per-field correctness vs gold labels | Catches enum\u002Ftype drift a valid-rate hides |\n| **Tool-selection accuracy** | right tool chosen for the intent | Drops as toolset grows → triggers routing |\n| **Arg accuracy** | tool args correct vs expected | Separate from selecting the right tool |\n| **Hallucinated-call rate** | calls to nonexistent tools \u002F invalid args | Should trend to ~0 with good contracts |\n\n**Do:** keep a golden set of inputs with expected structured outputs; assert per-field, not just \"did it parse.\" A 100% valid-rate with 60% field accuracy is a worse failure than a parse error — it *looks* fine and ships wrong data. **Don't:** measure only valid-rate; constrained decoding can make it 100% while the *contents* are nonsense.\n\n---\n\n## Quick reference — decision order\n\n1. Can I use **constrained decoding** for this schema? → Do it. Malformed JSON gone.\n2. **Validate** the result at the boundary with Zod\u002FPydantic into a typed object — always, even after #1.\n3. On failure: **repair** (≤2×, feed the exact error) → **escalate** model → **fallback** to a typed default\u002Fhuman.\n4. For tools: tight **contracts**, **validate args before executing**, catch **hallucinated** calls by not trusting them, **cap\u002Froute** when tools are many.\n5. Make every write **idempotent**; gate destructive actions on confirmation.\n6. **Eval** valid-rate *and* field accuracy continuously — the second is the one that lies quietly.\n",{"name":5,"tags":23,"description":7},[24,25,26,27],"ai-engineering","structured-output","function-calling","json-schema","main","4911ae44bbb35beef4511ba298475ba5a82b7e1c","ai-engineering-skills\u002Fstructured-output-and-tool-calling",0,"https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta","https:\u002F\u002Fdocs.vanta.theft.studio","https:\u002F\u002Fgithub.com\u002Fjpoindexter","jason.theft.studio","Just some guy trying to build stuff","2026-11-12 17:11:35",{"source":8,"skillName":5,"commands":39,"prompt":49,"zipAvailable":50},[40,43,46],{"tool":41,"command":42},"npx","npx skills add jpoindexter\u002FVanta --skill structured-output-and-tool-calling",{"tool":44,"command":45},"bunx","bunx skills add jpoindexter\u002FVanta --skill structured-output-and-tool-calling",{"tool":47,"command":48},"pnpm","pnpm dlx skills add jpoindexter\u002FVanta --skill structured-output-and-tool-calling","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fai-engineering-skills\u002Fstructured-output-and-tool-calling\u002FSKILL.md",true,[],[53,60,67,74,81,87],{"skillNo":54,"name":55,"nameZh":6,"description":56,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":18},"01M7582PEM7XM8B98VFRDV0C0J","write-changelog","Turn a range of commits into a user-facing changelog","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fopenclaw\u002Fskills\u002Fwrite-changelog\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":18},"01M7582KTQJMP5Y7GS83MS1C7Z","debug-flaky-test","Reproduce and isolate a flaky test before fixing it","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fopenclaw\u002Fskills\u002Fdebug-flaky-test\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":18},"01M7582HG7XFM4Z5NGG1YFECT8","triage-incident","First 10 minutes of an incident — stabilize, then diagnose","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fhermes\u002Fskills\u002Ftriage-incident\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":18},"01M7582DQABK7A96XHMB770KQZ","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M7582BBK8H0HGSXKK08FCXJ1","vanta-port-adapter","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Fvanta-port-adapter\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M75828P8PX5XSBMJ2BVVDA9Y","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Ftest-driven-development\u002FSKILL.md",[],[]]