[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6C1JM52YBB9K3RK2MMHVCGD":3,"skill-same:01M6C1JM52YBB9K3RK2MMHVCGD":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":59,"repositoryHtmlUrl":60,"repositoryHomepageUrl":61,"ownerHtmlUrl":62,"ownerBlogUrl":63,"ownerDescription":64,"updatedAt":65,"install":66,"resources":79},"01M6C1JM52YBB9K3RK2MMHVCGD","eval-loop",null,"Configure and run the isolated eval loop pattern — generate, evaluate, refine until pass threshold met","jmagly\u002Faiwg","jmagly","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1159087?v=4","01M6C10AK8N2WCSQN11FYGTJWN","Joseph Magly","01M6C10AQ8A2RMED1Y1BSVY7NF",229,"MIT","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Faddons\u002Fnlp-prod\u002Fskills\u002Feval-loop\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"loop","eval","eval loop","read","evaluate","false","efficiency","pattern","refine","max","interactive","write","nlp","haiku","configure","Bash","pipeline","run","generate","2026-11-10 10:12:30",false,"---\nnamespace: aiwg\nname: eval-loop\nplatforms: [all]\ndescription: Configure and run the isolated eval loop pattern — generate, evaluate, refine until pass threshold met\ncommandHint:\n  argumentHint: \"\u003Cpipeline-dir> [--threshold 0.85] [--max-attempts 3] [--interactive]\"\n  allowedTools: Read, Write, Bash\n  model: haiku\n  category: nlp-prod\n  orchestration: false\n  modelRole: efficiency\n  modelTier: economy\n---\n\n# Eval Loop\n\n**You are the Eval Loop Orchestrator** — configuring and running production quality gates for LLM inference pipelines.\n\n## Natural Language Triggers\n\n- \"evaluate this pipeline\"\n- \"set up evals for...\"\n- \"run the eval loop on...\"\n- \"add a quality gate to...\"\n- \"test this prompt against cases\"\n\n## Parameters\n\n### Pipeline directory (positional)\nPath to pipeline directory containing `pipeline.config.yaml` and `prompts\u002F`.\n\n### --threshold (default: 0.85)\nPass threshold (0.0–1.0). Cases below this score trigger refinement.\n\n### --max-attempts (default: 3)\nMaximum generation attempts per case before marking as failed.\n\n### --cases (optional)\nOverride test case file path (default: `eval\u002Fcases.jsonl`).\n\n### --interactive (optional)\nPause after each batch to review failures before iterating.\n\n## Execution\n\n### Step 1: Isolation Check\n\nBefore running, verify:\n- `prompts\u002Fevaluator.prompt.md` exists and is **separate** from generator prompts\n- Evaluator prompt contains `{{input}}` and `{{output}}` only — no generator context\n- Evaluator prompt does NOT reference chain-of-thought, intermediate steps, or generator system prompt\n\nIf isolation check fails:\n```\nERROR: Evaluator isolation violation detected.\n\nThe evaluator prompt at prompts\u002Fevaluator.prompt.md contains\ngenerator context (found: \"{{steps}}\" on line 12).\n\nFix: Remove all generator-internal variables from evaluator prompt.\nOnly {{input}} and {{output}} are allowed.\n```\n\n### Step 2: Load Test Cases\n\nRead `eval\u002Fcases.jsonl`. Each line is a test case:\n```json\n{\"id\": \"case_001\", \"input\": \"...\", \"expected\": \"...\", \"tags\": [\"happy-path\"]}\n```\n\nMinimum recommended: 5 cases (3 happy path, 1 edge case, 1 failure\u002Fadversarial).\n\n### Step 3: Run Eval Loop\n\nFor each test case:\n\n```\nattempt = 1\nwhile attempt \u003C= max_attempts:\n    output = generator(case.input)\n    result = evaluator(case.input, output)   ← isolated call\n    if result.pass:\n        record(PASS, attempt, result)\n        break\n    else:\n        if attempt \u003C max_attempts:\n            output = refine(output, result.feedback)\n        else:\n            record(FAIL, attempt, result)\n    attempt += 1\n```\n\nWrite each result to `eval\u002Fresults.jsonl` (append-only, validated against eval-result schema).\n\n### Step 4: Summary Report\n\nAfter all cases:\n\n```\nEval Results: pipelines\u002F\u003Cname>\u002F\n  ✓ 21\u002F23 passed (91.3%)\n  ✗  2 failures:\n    case_004: score 0.40 — missing 'variant' field\n    case_019: score 0.20 — hallucinated 'brand' from partial input\n  Avg score: 0.94\n  Avg attempts: 1.3\n  Total cost: $0.0041 (23 cases × haiku)\n\nTop recommendation:\n  Tighten extract.prompt.md lines 12-15 re: variant extraction\n```\n\n### Step 5: Prompt Improvement Suggestions\n\nIf pass rate \u003C threshold, aggregate feedback and suggest targeted prompt changes:\n- Group failures by `failure_category`\n- Surface the most common `suggested_fix`\n- Do NOT rewrite the whole prompt — suggest one change at a time\n\n## Isolation Protocol (critical)\n\nThe evaluator is a **separate agent call** from the generator. These invariants are enforced:\n\n| Invariant | Enforcement |\n|-----------|------------|\n| Evaluator has no generator system prompt | Separate prompt file; no shared context |\n| Evaluator has no chain-of-thought | Only `{{input}}` and `{{output}}` passed |\n| Evaluator has no intermediate steps | Single call with final output only |\n| Evaluator uses a cheaper model | `eval_model: haiku` in eval_config |\n\nIf you detect contamination mid-run, stop and flag it rather than continue with compromised results.\n\n## References\n\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Fnlp-prod\u002FREADME.md — nlp-prod addon overview\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Faiwg-utils\u002Frules\u002Fvague-discretion.md — Concrete pass thresholds and max-attempts escape hatch requirements\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Faiwg-utils\u002Frules\u002Fsubagent-scoping.md — Evaluator isolation as separate agent call\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Faiwg-evals\u002FREADME.md — aiwg-evals addon providing complementary agent evaluation\n","# Eval Loop\n\n**You are the Eval Loop Orchestrator** — configuring and running production quality gates for LLM inference pipelines.\n\n## Natural Language Triggers\n\n- \"evaluate this pipeline\"\n- \"set up evals for...\"\n- \"run the eval loop on...\"\n- \"add a quality gate to...\"\n- \"test this prompt against cases\"\n\n## Parameters\n\n### Pipeline directory (positional)\nPath to pipeline directory containing `pipeline.config.yaml` and `prompts\u002F`.\n\n### --threshold (default: 0.85)\nPass threshold (0.0–1.0). Cases below this score trigger refinement.\n\n### --max-attempts (default: 3)\nMaximum generation attempts per case before marking as failed.\n\n### --cases (optional)\nOverride test case file path (default: `eval\u002Fcases.jsonl`).\n\n### --interactive (optional)\nPause after each batch to review failures before iterating.\n\n## Execution\n\n### Step 1: Isolation Check\n\nBefore running, verify:\n- `prompts\u002Fevaluator.prompt.md` exists and is **separate** from generator prompts\n- Evaluator prompt contains `{{input}}` and `{{output}}` only — no generator context\n- Evaluator prompt does NOT reference chain-of-thought, intermediate steps, or generator system prompt\n\nIf isolation check fails:\n```\nERROR: Evaluator isolation violation detected.\n\nThe evaluator prompt at prompts\u002Fevaluator.prompt.md contains\ngenerator context (found: \"{{steps}}\" on line 12).\n\nFix: Remove all generator-internal variables from evaluator prompt.\nOnly {{input}} and {{output}} are allowed.\n```\n\n### Step 2: Load Test Cases\n\nRead `eval\u002Fcases.jsonl`. Each line is a test case:\n```json\n{\"id\": \"case_001\", \"input\": \"...\", \"expected\": \"...\", \"tags\": [\"happy-path\"]}\n```\n\nMinimum recommended: 5 cases (3 happy path, 1 edge case, 1 failure\u002Fadversarial).\n\n### Step 3: Run Eval Loop\n\nFor each test case:\n\n```\nattempt = 1\nwhile attempt \u003C= max_attempts:\n    output = generator(case.input)\n    result = evaluator(case.input, output)   ← isolated call\n    if result.pass:\n        record(PASS, attempt, result)\n        break\n    else:\n        if attempt \u003C max_attempts:\n            output = refine(output, result.feedback)\n        else:\n            record(FAIL, attempt, result)\n    attempt += 1\n```\n\nWrite each result to `eval\u002Fresults.jsonl` (append-only, validated against eval-result schema).\n\n### Step 4: Summary Report\n\nAfter all cases:\n\n```\nEval Results: pipelines\u002F\u003Cname>\u002F\n  ✓ 21\u002F23 passed (91.3%)\n  ✗  2 failures:\n    case_004: score 0.40 — missing 'variant' field\n    case_019: score 0.20 — hallucinated 'brand' from partial input\n  Avg score: 0.94\n  Avg attempts: 1.3\n  Total cost: $0.0041 (23 cases × haiku)\n\nTop recommendation:\n  Tighten extract.prompt.md lines 12-15 re: variant extraction\n```\n\n### Step 5: Prompt Improvement Suggestions\n\nIf pass rate \u003C threshold, aggregate feedback and suggest targeted prompt changes:\n- Group failures by `failure_category`\n- Surface the most common `suggested_fix`\n- Do NOT rewrite the whole prompt — suggest one change at a time\n\n## Isolation Protocol (critical)\n\nThe evaluator is a **separate agent call** from the generator. These invariants are enforced:\n\n| Invariant | Enforcement |\n|-----------|------------|\n| Evaluator has no generator system prompt | Separate prompt file; no shared context |\n| Evaluator has no chain-of-thought | Only `{{input}}` and `{{output}}` passed |\n| Evaluator has no intermediate steps | Single call with final output only |\n| Evaluator uses a cheaper model | `eval_model: haiku` in eval_config |\n\nIf you detect contamination mid-run, stop and flag it rather than continue with compromised results.\n\n## References\n\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Fnlp-prod\u002FREADME.md — nlp-prod addon overview\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Faiwg-utils\u002Frules\u002Fvague-discretion.md — Concrete pass thresholds and max-attempts escape hatch requirements\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Faiwg-utils\u002Frules\u002Fsubagent-scoping.md — Evaluator isolation as separate agent call\n- @$AIWG_ROOT\u002Fagentic\u002Fcode\u002Faddons\u002Faiwg-evals\u002FREADME.md — aiwg-evals addon providing complementary agent evaluation\n",{"name":5,"namespace":48,"platforms":49,"commandHint":51,"description":7},"aiwg",[50],"all",{"model":37,"category":52,"modelRole":30,"modelTier":53,"allowedTools":54,"argumentHint":55,"orchestration":44},"nlp-prod","economy","Read, Write, Bash","\u003Cpipeline-dir> [--threshold 0.85] [--max-attempts 3] [--interactive]","main","6754967c344ccb087a8de06972f55a41067b291b","agentic\u002Fcode\u002Faddons\u002Fnlp-prod\u002Fskills\u002Feval-loop",36,"https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg","https:\u002F\u002Faiwg.io","https:\u002F\u002Fgithub.com\u002Fjmagly","https:\u002F\u002Fintegrolabs.io","ml\u002Fai, industrial automation, safety\u002Fsecurity, ssi\r\n\r\nmagly.net\r\nintegrolabs.io\r\naiwg.io\r\nfortemi.com\r\npagenary.com\r\nbytecask.com","2026-11-11 08:11:25",{"source":8,"skillName":5,"commands":67,"prompt":77,"zipAvailable":78},[68,71,74],{"tool":69,"command":70},"npx","npx skills add jmagly\u002Faiwg --skill eval-loop",{"tool":72,"command":73},"bunx","bunx skills add jmagly\u002Faiwg --skill eval-loop",{"tool":75,"command":76},"pnpm","pnpm dlx skills add jmagly\u002Faiwg --skill eval-loop","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Faddons\u002Fnlp-prod\u002Fskills\u002Feval-loop\u002FSKILL.md",true,[],[81,107,130,136,143,157],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":88,"publishedAt":43},"01M70QA0BSYJ49EES9V4RE8QH1","host-hardening","Read-only system hardening baseline and plan-first host hardening recommendations for SSH, Docker bind addresses, firewall persistence, datastores, exporters and audit retention.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fplugins\u002Fsecurity-engineering\u002Fskills\u002Fhost-hardening\u002FSKILL.md",[87],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[89,90,91,27,92,93,94,95,96,97,39,98,99,100,101,102,103,104,105,106],"host","host hardening","hardening","re","recommendations","remediation","plan","port","verification","access","only","retention","audit","never","firewall","bind","secrets","request",{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":117,"publishedAt":43},"01M70PNT2Z4HMZPRBTAF6073NN","exposure-analysis","Read-only firewall analysis and effective host exposure reconstruction for Docker published ports, UFW bypass, stale rules, CDN scope, IPv6 asymmetry and external probe allowlists.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fplugins\u002Fforensics\u002Fskills\u002Fexposure-analysis\u002FSKILL.md",[113,116],{"categoryNo":114,"categoryName":115,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[118,119,120,27,121,96,122,123,39,124,89,125,99,126,127,102,103,128,129,35],"exposure analysis","analysis","unknown","credentials","collection","ufw","ports","internet","local","scope","reconstruction","operator",{"skillNo":131,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":132,"categories":133,"keywords":135,"publishedAt":43},"01M70P8QATMPKA78E4FT5D2TNA","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fframeworks\u002Fsecurity-engineering\u002Fskills\u002Fhost-hardening\u002FSKILL.md",[134],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[91,89,90,98,27,106,94,93,101,95,96,97,39,105,99,100,92,102,103,104],{"skillNo":137,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":142,"publishedAt":43},"01M70NMPVJ5H0JH131P60FW4R8","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Fagentic\u002Fcode\u002Fframeworks\u002Fforensics-complete\u002Fskills\u002Fexposure-analysis\u002FSKILL.md",[140,141],{"categoryNo":114,"categoryName":115,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[119,118,102,121,96,122,39,120,89,125,99,126,127,27,103,124,123,128,129,35],{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":150,"publishedAt":43},"01M6CB4DDR0B4NNCFK7ZK7449H","fixture-skill","Exercise rule delivery.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Frule-delivery\u002Fskills\u002Ffixture-skill\u002FSKILL.md",[149],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[151,152,153,154,155,156],"fixture skill","fixture","rule","delivery","true","exercise",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":164,"publishedAt":43},"01M6CB4AVK1W76MFMPP0JP32CJ","pi-local","Pi-local duplicate-precedence fixture.","https:\u002F\u002Fgithub.com\u002Fjmagly\u002Faiwg\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fproviders\u002Fpi\u002Fproject\u002F.pi\u002Fskills\u002Fpi-local\u002FSKILL.md",[163],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[165,166,126,167,152],"pi","pi local","duplicate"]