[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7F52S60V72F2W44SN0RWRN3":3,"skill-same:01M7F52S60V72F2W44SN0RWRN3":45},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":14,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M7F52S60V72F2W44SN0RWRN3","evaluation-harness-design",null,"Methodology for defining retrieval quality metrics, acceptance thresholds, and benchmarking frameworks before tooling selection.","cwik-tech\u002Foctopus","cwik-tech","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F54542292?v=4","01M7F42E123GEHVK6RWC5MSWJ9","Tomasz Cwik","01M7F42E1PD4P54MTG8PG32VCV",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fagents\u002Fretrieval-architect\u002Fskills\u002Fevaluation-harness-design\u002FSKILL.md",[],[],"2026-11-16 10:05:34",false,"---\nname: evaluation-harness-design\ndescription: Methodology for defining retrieval quality metrics, acceptance thresholds, and benchmarking frameworks before tooling selection.\n---\n\n# Evaluation Harness Design\n\n## Purpose\n\nDefine how to measure retrieval quality systematically. Choose metrics that matter for agent use cases. Set acceptance thresholds that reflect real requirements. Design a benchmarking workflow so the team can re-run evaluations as retrieval changes without redoing the entire testing process.\n\n## When to Use\n\n- Designing evaluation frameworks before retrieval architecture is chosen\n- Establishing quality gates for retrieval systems\n- Setting up continuous evaluation as retrieval evolves\n- Defining acceptance criteria for production deployment\n\n## Core Metrics\n\n### Recall@k\n\n**What it measures:** Of the top k results returned, how many relevant passages did we actually retrieve?\n\nRecall@10 = (# relevant passages in top 10) \u002F (# total relevant passages)\n\n- Answers the question: \"Will the agent find an answer if it looks at the top k results?\"\n- Common thresholds: Recall@10 ≥ 80%, Recall@5 ≥ 60%\n- For FST: Multiple valid passages often answer one question. If the corpus has redundancy, Recall@10 might be your primary metric.\n- **Interpretation risk:** High Recall doesn't guarantee the agent picks the *best* passage; reranking and citation correctness matter.\n\n### nDCG \u002F MRR (Ranking Quality)\n\n**nDCG (normalized Discounted Cumulative Gain):** How well are relevant results ranked? Do the best answers appear first?\n\nnDCG@10 = (DCG@10) \u002F (Ideal DCG@10)\n\n- DCG gives higher credit to relevant results ranked earlier\n- Answers: \"Are the top results better than the bottom results?\"\n- Common thresholds: nDCG@10 ≥ 0.7 (70%)\n- For FST: Useful if some passages are more authoritative or complete than others. If all correct answers are equally valid, nDCG matters less than Recall.\n\n**MRR (Mean Reciprocal Rank):** How high is the first relevant result, on average?\n\nMRR = 1 \u002F (average rank of first relevant result)\n\n- Simpler than nDCG; good when you care most about the first correct answer\n- Common threshold: MRR ≥ 0.6\n\n### Citation Correctness\n\n**What it measures:** For every passage returned, can we trace it to a source document and locate it in that document?\n\nCitation correctness = (passages with valid source + chunk ID + location) \u002F (total passages returned)\n\n- Answers: \"Is every answer traceable to the corpus?\"\n- Common threshold: 100% (no citations should be wrong)\n- **Critical for FST:** Agents must cite sources. A retrieval system that returns high-quality passages but loses citation metadata is unusable.\n\nMeasure:\n- Source document ID present?\n- Chunk ID or heading path present?\n- Physical location (page, section) reconstructable?\n- Can a human verify the cited passage matches the source?\n\nIf any citation is wrong, the entire response is untrustworthy.\n\n### Latency (Response Time)\n\n**What it measures:** How long does a query take?\n\n- p50 (median): 50% of queries complete in this time\n- p95 (95th percentile): 95% of queries complete in this time\n- Common thresholds: p50 ≤ 200ms, p95 ≤ 1000ms (agent-facing SLA)\n\nFor FST: Determine from actual agent workflows. If an agent waits on retrieval, slow retrieval blocks the entire interaction.\n\n### Cost per Query\n\n**What it measures:** How much does one retrieval query cost?\n\n- Embedding API calls (cost per query)\n- Reranking (cost per top-k result)\n- Storage and indexing (amortized per query)\n\nCommon thresholds: Set based on query volume and budget. If the corpus answers 1,000 queries\u002Fday, total daily cost must fit in operational budget.\n\n## Setting Acceptance Thresholds\n\nDon't choose thresholds in abstract. Ground them in requirements:\n\n| Requirement | Threshold Implication |\n|---|---|\n| Agents must find an answer 95% of the time | Recall@10 ≥ 90% |\n| If multiple answers exist, show the best one first | nDCG@10 ≥ 0.75 |\n| Every cited passage must be verifiable | Citation correctness = 100% |\n| Agents cannot wait >1 second per query | p95 latency ≤ 1000ms |\n| Monthly retrieval cost must fit in $X budget | Cost per query ≤ $(X \u002F monthly queries) |\n\nThresholds should be tight enough to catch real failures, loose enough that the team can actually meet them.\n\n## Benchmarking Workflow\n\nDesign the workflow so evaluations can be repeated as retrieval evolves:\n\n1. **Baseline:** Run the evaluation set against current retrieval. Record all metrics. This is your reference point.\n2. **Change:** Modify chunking, embedding model, reranker, or metadata filters.\n3. **Re-evaluate:** Run the same evaluation set against the modified retrieval. Compare metrics to baseline.\n4. **Decision:** Did the change improve recall, latency, cost? By how much? Is it worth the tradeoff?\n\n### Automation Checklist\n\n- [ ] Evaluation set loaded and versioned (tag with corpus version, date)\n- [ ] Retrieval queries automated (script that runs all questions against current retrieval)\n- [ ] Metrics computed automatically (Recall@k, nDCG, citation correctness, latency, cost)\n- [ ] Results saved with timestamp, corpus version, retrieval configuration\n- [ ] Comparison tool shows delta from baseline\n- [ ] Runbook for re-running evaluations documented\n\nWithout automation, evaluation becomes ad-hoc and people skip it.\n\n## Testing Hybrid Retrieval\n\nA common hypothesis: hybrid (dense + lexical) retrieval beats either alone.\n\nTest this systematically:\n\n| Strategy | Queries | Recall@10 | nDCG@10 | Citation Correct | Latency p95 | Cost per Query |\n|---|---|---|---|---|---|---|\n| Lexical only (BM25) | 200 | 72% | 0.68 | 100% | 45ms | $0.001 |\n| Dense only (embeddings) | 200 | 78% | 0.71 | 100% | 120ms | $0.015 |\n| Hybrid (combined) | 200 | 85% | 0.76 | 100% | 140ms | $0.016 |\n| Hybrid + reranker | 200 | 87% | 0.78 | 100% | 180ms | $0.020 |\n\nConclusions:\n- Hybrid beats either alone (85% vs 72–78%)\n- Reranker adds 2–3% recall, but cost and latency increase 30%\n- Cost per query (hybrid + reranker) is ~20x lexical alone\n\nDecision: \"Hybrid retrieval is justified. Reranking is marginal; proceed with hybrid, skip reranker for now.\"\n\n## Quality Gates\n\nEstablish gates that block decisions:\n\n| Gate | Blocks What | Trigger |\n|---|---|---|\n| Citation correctness \u003C 100% | Agent API finalization | Wrong citations are unacceptable |\n| Recall@10 \u003C acceptance threshold | Architecture lock-in | Retrieval doesn't meet requirements |\n| Latency p95 > SLA | Production deployment | Speed commitments not met |\n| Cost per query > budget | Scaling decision | Economics don't work at volume |\n\nIf a gate fails, the team does not proceed. They tune retrieval or revisit requirements.\n\n## Output\n\nA documented evaluation harness including:\n- **Metric definitions:** Formulas and implementation for Recall@k, nDCG, citation correctness, latency, cost\n- **Acceptance thresholds:** Per-metric targets with justification\n- **Benchmarking workflow:** Steps and automation for re-running evaluations\n- **Baseline results:** Metrics for current\u002Freference retrieval (if any) against the evaluation set\n- **Quality gates:** Conditions that block architecture, deployment, or scaling decisions\n- **Reporting template:** How results are presented (tables, charts, narrative)\n\nThe harness becomes a living artifact. Update it as requirements change. Document every gate that blocks a decision.\n","# Evaluation Harness Design\n\n## Purpose\n\nDefine how to measure retrieval quality systematically. Choose metrics that matter for agent use cases. Set acceptance thresholds that reflect real requirements. Design a benchmarking workflow so the team can re-run evaluations as retrieval changes without redoing the entire testing process.\n\n## When to Use\n\n- Designing evaluation frameworks before retrieval architecture is chosen\n- Establishing quality gates for retrieval systems\n- Setting up continuous evaluation as retrieval evolves\n- Defining acceptance criteria for production deployment\n\n## Core Metrics\n\n### Recall@k\n\n**What it measures:** Of the top k results returned, how many relevant passages did we actually retrieve?\n\nRecall@10 = (# relevant passages in top 10) \u002F (# total relevant passages)\n\n- Answers the question: \"Will the agent find an answer if it looks at the top k results?\"\n- Common thresholds: Recall@10 ≥ 80%, Recall@5 ≥ 60%\n- For FST: Multiple valid passages often answer one question. If the corpus has redundancy, Recall@10 might be your primary metric.\n- **Interpretation risk:** High Recall doesn't guarantee the agent picks the *best* passage; reranking and citation correctness matter.\n\n### nDCG \u002F MRR (Ranking Quality)\n\n**nDCG (normalized Discounted Cumulative Gain):** How well are relevant results ranked? Do the best answers appear first?\n\nnDCG@10 = (DCG@10) \u002F (Ideal DCG@10)\n\n- DCG gives higher credit to relevant results ranked earlier\n- Answers: \"Are the top results better than the bottom results?\"\n- Common thresholds: nDCG@10 ≥ 0.7 (70%)\n- For FST: Useful if some passages are more authoritative or complete than others. If all correct answers are equally valid, nDCG matters less than Recall.\n\n**MRR (Mean Reciprocal Rank):** How high is the first relevant result, on average?\n\nMRR = 1 \u002F (average rank of first relevant result)\n\n- Simpler than nDCG; good when you care most about the first correct answer\n- Common threshold: MRR ≥ 0.6\n\n### Citation Correctness\n\n**What it measures:** For every passage returned, can we trace it to a source document and locate it in that document?\n\nCitation correctness = (passages with valid source + chunk ID + location) \u002F (total passages returned)\n\n- Answers: \"Is every answer traceable to the corpus?\"\n- Common threshold: 100% (no citations should be wrong)\n- **Critical for FST:** Agents must cite sources. A retrieval system that returns high-quality passages but loses citation metadata is unusable.\n\nMeasure:\n- Source document ID present?\n- Chunk ID or heading path present?\n- Physical location (page, section) reconstructable?\n- Can a human verify the cited passage matches the source?\n\nIf any citation is wrong, the entire response is untrustworthy.\n\n### Latency (Response Time)\n\n**What it measures:** How long does a query take?\n\n- p50 (median): 50% of queries complete in this time\n- p95 (95th percentile): 95% of queries complete in this time\n- Common thresholds: p50 ≤ 200ms, p95 ≤ 1000ms (agent-facing SLA)\n\nFor FST: Determine from actual agent workflows. If an agent waits on retrieval, slow retrieval blocks the entire interaction.\n\n### Cost per Query\n\n**What it measures:** How much does one retrieval query cost?\n\n- Embedding API calls (cost per query)\n- Reranking (cost per top-k result)\n- Storage and indexing (amortized per query)\n\nCommon thresholds: Set based on query volume and budget. If the corpus answers 1,000 queries\u002Fday, total daily cost must fit in operational budget.\n\n## Setting Acceptance Thresholds\n\nDon't choose thresholds in abstract. Ground them in requirements:\n\n| Requirement | Threshold Implication |\n|---|---|\n| Agents must find an answer 95% of the time | Recall@10 ≥ 90% |\n| If multiple answers exist, show the best one first | nDCG@10 ≥ 0.75 |\n| Every cited passage must be verifiable | Citation correctness = 100% |\n| Agents cannot wait >1 second per query | p95 latency ≤ 1000ms |\n| Monthly retrieval cost must fit in $X budget | Cost per query ≤ $(X \u002F monthly queries) |\n\nThresholds should be tight enough to catch real failures, loose enough that the team can actually meet them.\n\n## Benchmarking Workflow\n\nDesign the workflow so evaluations can be repeated as retrieval evolves:\n\n1. **Baseline:** Run the evaluation set against current retrieval. Record all metrics. This is your reference point.\n2. **Change:** Modify chunking, embedding model, reranker, or metadata filters.\n3. **Re-evaluate:** Run the same evaluation set against the modified retrieval. Compare metrics to baseline.\n4. **Decision:** Did the change improve recall, latency, cost? By how much? Is it worth the tradeoff?\n\n### Automation Checklist\n\n- [ ] Evaluation set loaded and versioned (tag with corpus version, date)\n- [ ] Retrieval queries automated (script that runs all questions against current retrieval)\n- [ ] Metrics computed automatically (Recall@k, nDCG, citation correctness, latency, cost)\n- [ ] Results saved with timestamp, corpus version, retrieval configuration\n- [ ] Comparison tool shows delta from baseline\n- [ ] Runbook for re-running evaluations documented\n\nWithout automation, evaluation becomes ad-hoc and people skip it.\n\n## Testing Hybrid Retrieval\n\nA common hypothesis: hybrid (dense + lexical) retrieval beats either alone.\n\nTest this systematically:\n\n| Strategy | Queries | Recall@10 | nDCG@10 | Citation Correct | Latency p95 | Cost per Query |\n|---|---|---|---|---|---|---|\n| Lexical only (BM25) | 200 | 72% | 0.68 | 100% | 45ms | $0.001 |\n| Dense only (embeddings) | 200 | 78% | 0.71 | 100% | 120ms | $0.015 |\n| Hybrid (combined) | 200 | 85% | 0.76 | 100% | 140ms | $0.016 |\n| Hybrid + reranker | 200 | 87% | 0.78 | 100% | 180ms | $0.020 |\n\nConclusions:\n- Hybrid beats either alone (85% vs 72–78%)\n- Reranker adds 2–3% recall, but cost and latency increase 30%\n- Cost per query (hybrid + reranker) is ~20x lexical alone\n\nDecision: \"Hybrid retrieval is justified. Reranking is marginal; proceed with hybrid, skip reranker for now.\"\n\n## Quality Gates\n\nEstablish gates that block decisions:\n\n| Gate | Blocks What | Trigger |\n|---|---|---|\n| Citation correctness \u003C 100% | Agent API finalization | Wrong citations are unacceptable |\n| Recall@10 \u003C acceptance threshold | Architecture lock-in | Retrieval doesn't meet requirements |\n| Latency p95 > SLA | Production deployment | Speed commitments not met |\n| Cost per query > budget | Scaling decision | Economics don't work at volume |\n\nIf a gate fails, the team does not proceed. They tune retrieval or revisit requirements.\n\n## Output\n\nA documented evaluation harness including:\n- **Metric definitions:** Formulas and implementation for Recall@k, nDCG, citation correctness, latency, cost\n- **Acceptance thresholds:** Per-metric targets with justification\n- **Benchmarking workflow:** Steps and automation for re-running evaluations\n- **Baseline results:** Metrics for current\u002Freference retrieval (if any) against the evaluation set\n- **Quality gates:** Conditions that block architecture, deployment, or scaling decisions\n- **Reporting template:** How results are presented (tables, charts, narrative)\n\nThe harness becomes a living artifact. Update it as requirements change. Document every gate that blocks a decision.\n",{"name":5,"description":7},"main","f0f0efe1d4c64b51a51f44427ad147750d29154e","agents\u002Fretrieval-architect\u002Fskills\u002Fevaluation-harness-design","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus","www.cuteoctopus.site","https:\u002F\u002Fgithub.com\u002Fcwik-tech","2026-10-27 06:12:19",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add cwik-tech\u002Foctopus --skill evaluation-harness-design",{"tool":37,"command":38},"bunx","bunx skills add cwik-tech\u002Foctopus --skill evaluation-harness-design",{"tool":40,"command":41},"pnpm","pnpm dlx skills add cwik-tech\u002Foctopus --skill evaluation-harness-design","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fagents\u002Fretrieval-architect\u002Fskills\u002Fevaluation-harness-design\u002FSKILL.md",true,[],[46,53,60,67,74,81],{"skillNo":47,"name":48,"nameZh":6,"description":49,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":50,"categories":51,"keywords":52,"publishedAt":19},"01M7F5MQVQ3CHGM77Q51M4B1AC","twenty-year-land-plan","Apply the Bren discipline of evaluating every land and development decision on a twenty-year horizon, with a comprehensive plan for how the land, the community, and the surrounding public infrastructure evolve together. Deals optimized for a five-year exit are structurally different from deals that compound across generations; insist on knowing which kind you are making. Use when evaluating land, entitlements, master-planned development, or phasing decisions, and when separating five-year exit deals from generational holds.","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fskills\u002Ftwenty-year-land-plan\u002FSKILL.md",[],[],{"skillNo":54,"name":55,"nameZh":6,"description":56,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":19},"01M7F5MGQ8JSV27SFDQVVGFQE3","thematic-gut-check","Apply the Gray discipline of identifying the structural theme behind every real estate deal, testing where that theme sits in its capital-flow lifecycle, and insisting that deals without a named theme — or deals late in a crowded theme — clear a higher bar. A mediocre deal in a strong theme usually beats a clever deal in a weak one. Use when naming the structural thesis behind a deal, judging whether that theme is early or crowded, and setting the bar the deal must clear.","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fskills\u002Fthematic-gut-check\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M7F5M8V64Y0ZD8PTPWEJQXJ9","supply-pipeline-gut-check","Apply the Zell discipline of grounding every real estate thesis in the local supply pipeline. Demand counts over adjectives — units under construction, units permitted, units proposed — and convert them into a forward absorption schedule. A deal that ignores the pipeline is a bet on a market you have not actually studied. Use when validating a market thesis, forecasting absorption and rent growth, or counting units under construction, permitted, and proposed.","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fskills\u002Fsupply-pipeline-gut-check\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M7F5M3J2Q146BJC13TD3YCXF","ruin-scenario-mapping","Apply the Taleb discipline of identifying, for every real estate deal, the specific chain of events that produces ruin — permanent, unrecoverable capital loss — and verifying that no single such chain has non-trivial probability. Deals with any plausible ruin path must be restructured or abandoned. Use when sizing leverage, testing refinancing, covenant, and liquidity risk, or checking whether any single failure chain can permanently wipe out the position.","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fskills\u002Fruin-scenario-mapping\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7F5KZDS0DYD172P6HS0WXEM","revenue-cost-architecture","Design a venture's revenue streams, pricing logic, cost structure, and unit economics so the business model has a coherent theory of value capture, not just value creation. Use after the value proposition and customer segment are set and before financial projections, when choosing a pricing model, calculating contribution margin or CAC payback, or checking whether the unit economics hold at scale.","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fskills\u002Frevenue-cost-architecture\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7F5KP1CNQMYAVCDTV4EEW3J","replacement-cost-stress-test","Apply the Buffett discipline of pricing every real estate asset against the cost to rebuild it from scratch. The ratio of purchase price to replacement cost sets a ceiling on long-run downside: below replacement cost, new supply cannot profitably undercut you; at or above it, you are buying a liability disguised as an asset. Use when setting a purchase price, judging downside protection, or testing how exposed an asset is to newly built competing supply.","https:\u002F\u002Fgithub.com\u002Fcwik-tech\u002Foctopus\u002Fblob\u002Fmain\u002Fskills\u002Freplacement-cost-stress-test\u002FSKILL.md",[],[]]