deterministic-validation-loop
Architect, implement, and operate spec-first deterministic validation loops (3-tier validation, exact error injection, and bounded retries) for LLM structured outputs. Do not use for subjective or open-ended prose evaluation where deterministic truth functions cannot be written.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- deterministic-validation-loop
- description
- Architect, implement, and operate spec-first deterministic validation loops (3-tier validation, exact error injection, and bounded retries) for LLM structured outputs. Do not use for subjective or open-ended prose evaluation where deterministic truth functions cannot be written.
02 资源文件 3
- 参考文档CARD.md3.6 KB
- 参考文档config.default.yaml367 B
- 脚本scripts/validation_loop_engine.py16.7 KB
03 SKILL.md 正文
Deterministic Validation Loop: Spec-First Agent Output Reliability
The deterministic-validation-loop skill provides the definitive engineering methodology for ensuring absolute correctness when AI agents generate structured outputs (JSON schemas, deployment configurations, SQL queries, ASTs, and API payloads).
While popular tutorials propose LLM "reflection" turns to review structured outputs, reflection produces blind spot symmetry—where the reviewing model rubber-stamps invalid outputs as "looks good!", leaking broken state into production. This skill synthesizes Manish Ramavat's foundational literature (What to Do When Reflection Won't Fix Your AI Agent's Output, freeCodeCamp, 2026), enforcing a strict spec-first architecture, 3-tier deterministic validation hierarchy, exact machine-generated error injection, and bounded give-up triage.
[1. Spec & Validator Definition] -> [2. Graph & Node Topology] -> [3. Error Feedback Injection] -> [4. Bounded Budget & Triage] -> [5. The Visual Brief & Mandatory Checkpoint Gate]
See CARD.md for the companion summary card, stage matrix, and verification checklist. Consult self-evaluating-ai-pipeline for subjective/open-ended evaluation and crafting-skills for skill authoring standards.
1. Spec & Validator Definition (Spec-First Architecture)
Build the validator before writing agent prompts. The validator is your machine-checkable specification. It defines "correct" in unambiguous, executable terms.
- Implement the 3-Tier Validation Hierarchy:
- Tier 1 (Structural Syntax & Schema): Verify JSON parseability, presence of required attributes, and basic primitive types using
jsonschema.validate()or native parsers. Bail early on failure; never check business rules against unparsed or broken data structures. - Tier 2 (Boundary & Pattern Constraints): Verify numerical intervals (e.g.
1 <= replicas <= 20), memory/CPU limits, and regex format conformity (e.g.^[a-z][a-z0-9-]*$). - Tier 3 (Cross-Field Business Logic Invariants): Verify multi-attribute relational invariants (e.g.,
replicas > 5requirescpu_limit >= 1.0;timeout_seconds < interval_seconds). These are domain-specific invariants absent from model training priors.
- Tier 1 (Structural Syntax & Schema): Verify JSON parseability, presence of required attributes, and basic primitive types using
- Standardize Validator Contract:
- Expose an unambiguous signature:
def validate_payload(payload: dict) -> tuple[bool, list[str]]: """Returns (is_valid, list_of_error_strings).""" - Ensure errors are returned as human-readable, actionable strings targeting specific keys.
- Expose an unambiguous signature:
Completion criterion: Validator function authored, decoupled from LLM dependencies, executing all 3 tiers in <1ms, returning
(is_valid, errors).
2. Stateful Graph & Node Topology Synthesis
Wire the generator and validator into a cyclic state graph (e.g., using LangGraph or an in-process ReAct state engine).
- Declare State Schema:
from typing import TypedDict class ValidationState(TypedDict): request: str payload: dict | None errors: list[str] attempts: int - Construct Discrete Nodes:
generate_node(state): Calls the LLM to synthesize the payload. Whenstate["errors"]is populated, injects previous errors directly into the prompt. Incrementsattempts.validate_node(state): Runs the pure deterministic validator. Bails if payload isNone(JSON parse error) and updatesstate["errors"].
- Establish Deterministic Routing Edge:
route_condition(state) -> "done" | "retry":- Returns
"done"iflen(state["errors"]) == 0. - Returns
"retry"ifstate["attempts"] < 3. - Returns
"done"ifstate["attempts"] >= 3(escalating to failure triage).
- Returns
Completion criterion: State machine graph compiled with explicit generate, validate, and conditional retry edges terminating deterministically.
3. Unambiguous Error Feedback Injection
Differentiate between error detection (delegated to the deterministic validator) and error correction (delegated to the LLM).
- Zero Natural Language Critique:
- Never use vague conversational critique ("Please make sure the CPU limits are appropriate").
- Never ask an LLM to evaluate whether its own output meets the spec.
- Structured Delta Injection:
- When retrying, inject the exact list of machine-generated error messages directly into the user/human turn:
content = f"Generate configuration for: {state['request']}" if state["errors"]: content += "\n\nYour previous attempt had these errors:\n" content += "\n".join(f"- {err}" for err in state["errors"]) content += "\nFix ALL of them."
- When retrying, inject the exact list of machine-generated error messages directly into the user/human turn:
- Clean Prompt Hygiene:
- Maintain the static system prompt as a concise definition of the required fields. Do not bloat prompts with historical conversation banter.
Completion criterion: Generation node dynamically appends exact bulleted validator error strings on all retry iterations.
4. Bounded Budget & Give-Up Triage
Enforce a hard ceiling on retries. An agent that fails three consecutive attempts is suffering from prompt/spec ambiguity, not random generation noise.
- Cap Retries at Exactly 3 Attempts:
- Prohibit unbounded retry loops ($>3$ attempts). Attempting 4+ times burns API tokens with near-zero marginal recovery.
- Execute Three-Way Triage on Failure:
- Telemetry Logging: Record the original request, raw invalid payload, and final error list to a dedicated observability sink. This forms the primary dataset for refining ambiguous prompts or conflicting schemas.
- Typed Rejection (Fail Closed): Reject the request with an explicit error response (e.g., HTTP 422 Unprocessable Entity with error list). Never allow unvalidated or fallback-default payloads to silently leak downstream.
- Human Escalation: For critical operational paths (deployment pipelines, database migrations), route the failed state to a human supervisor queue.
Completion criterion: Graph halts at
attempts == 3with structured error logging and typed rejection, preventing silent corruptions.
5. Applicability & Verification Gate
Audit the agent application to ensure the pattern is correctly applied and verified with automated test contracts.
- The Boolean Truth Test:
- Ask: Can you write a function returning
trueorfalsefor this agent output? - If yes → Enforce
deterministic-validation-loop. - If no → Route to semantic LLM evaluation (
self-evaluating-ai-pipeline) or human review.
- Ask: Can you write a function returning
- Test Contract Verification:
- Author characterization tests verifying:
- Valid payload passes on attempt 1.
- Invalid syntax/schema triggers immediate Tier 1 rejection.
- Boundary and cross-field errors trigger retry with injected errors and pass on attempt 2.
- Persistent errors halt cleanly at attempt 3 with logged triage.
- Author characterization tests verifying:
Completion criterion: Test suite asserts passing payloads, recovery on retry, and clean failure closure at attempt 3 with zero unhandled exceptions.
6. Micro-Kernel IoC & CLI Seams
The deterministic validation loop is elevated into the Brain Harness micro-kernel as an authoritative service and CLI:
- Domain Engine: validation_loop_engine.py provides slotted, frozen models (
ValidationSpec,ValidationRule,DeterministicValidationEngine) with sub-millisecond execution and built-in preset specs (deployment_config,plugin_manifest). - IoC Service Key:
DETERMINISTIC_VALIDATION_SERVICE_KEYexportsServiceKey[DeterministicValidationService]for in-memory resolution (context.require()). - Plugin Packaging: deterministic_validation_loop plugin registers the service and exposes agent tools (
validate_payload,run_validation_loop). - Headless Click CLI: Execute validation from terminal or CI scripts:
harness validation-loop specsharness validation-loop validate --spec deployment_config --file config.jsonharness validation-loop run --request "Deploy service" --spec deployment_configharness validation-loop brief --output brief.html
Anti-Patterns
- Blind Spot Symmetry — Asking an LLM to critique another LLM's structured output. The identical weights and priors that missed the constraint during generation will approve the broken output during review.
- Invisible False Positives — Relying on reflection steps that say "looks good!" on broken JSON, allowing corrupt configurations to silently reach production.
- Infinite Token Burning — Allowing loops to retry 5-10 times hoping for luck, rather than halting at 3 attempts and escalating spec ambiguity.
- Spec-Last Inversion — Writing agent system prompts before writing the validator function. The validator is the specification.
- Silent Leakage — Falling through to downstream execution on exhausted retries without raising an error or marking state as invalid.
- Vague Critique Injection — Feeding conversational feedback ("please check your numbers") instead of exact machine-generated error strings.
04 同仓库更多 Skills
ui-behavior-primitive-architect
Architect, audit, and verify zero-dependency headless UI behavior primitives, WebMCP agent bridges, WAI-ARIA/SSR/RTL tri-modal checks, and framework expert routing. Do not use for static CSS theme styling or backend database migrations.
trustgraph-ontology-governor
Govern knowledge graphs, SHACL-AF triple policies, OntologyRAG subclass validation, PROV-O dual named graphs, and Laya entropy decision scoring. Do not use for relational SQL queries or generic web scraping.
shim-harness-governor
Govern universal CLI agent loops, 4-tier tool permissions, ancestor-symlink workspace sandboxes, deterministic-first command routing, collision-safe MCP namespacing, and layered skill loading. Do not use for static UI styling or external SQL analytics.
graphiti-zep-memory-governor
Govern bi-temporal knowledge graphs with Graphiti future-edge clamping, MinHash/LSH entity dedup, RRF/MMR reranking, and Zep typed ontologies. Do not use for static SQL joins or generic web crawling.
agora-mcp-workbench-architect
Architect, audit, and orchestrate Agora Workbench MCP servers, loopback sidecars, unified state graphs, gateway/dispatcher connectors, and policy-enforced data lakes. Do not use for static frontend CSS styling or non-MCP desktop GUI automation.
skill-eval-assurance-runner
Evaluate and assure agent skills using small-N Wilson confidence intervals, unbiased pass^k reliability, deterministic trajectory graders, preflight boundary honesty, and sealed digest verification. Do not use for subjective prose review or unconstrained production deployment.