[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EVV74QWW0QRFFCD9MTP18Q":3,"skill-same:01M7EVV74QWW0QRFFCD9MTP18Q":64},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":30,"updatedAt":31,"install":32,"resources":45},"01M7EVV74QWW0QRFFCD9MTP18Q","advanced-evaluation",null,"This skill should be used when the user asks to \"implement LLM-as-judge\", \"compare model outputs\", \"create evaluation rubrics\", \"mitigate evaluation bias\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.","lza6\u002FArmor-piercing-system-prompts","lza6","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F128880206?v=4","01M7EB3K7WPX5T28EJ2NVHE6EG","Chinese-tingfeng","01M7EB3K97E1KQ16QWKJMFNGX8",11,"MIT","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F05-技能与代理开发\u002Fadvanced-evaluation\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: advanced-evaluation\ndescription: This skill should be used when the user asks to \"implement LLM-as-judge\", \"compare model outputs\", \"create evaluation rubrics\", \"mitigate evaluation bias\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building automated evaluation pipelines for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A\u002FB tests for prompt or model changes\n- Creating rubrics for human or automated evaluation\n- Analyzing correlation between automated and human judgments\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nSelect between two primary approaches based on whether ground truth exists:\n\n**Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.\n\n**Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Achieves higher human-judge agreement than direct scoring for preference tasks (Zheng et al., 2023). Watch for position bias and length bias.\n\n### The Bias Landscape\n\nMitigate these systematic biases in every evaluation system:\n\n**Position Bias**: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.\n\n**Length Bias**: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.\n\n**Verbosity Bias**: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.\n\n### Metric Selection Framework\n\nMatch metrics to the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass\u002Ffail) | Recall, Precision, F1 | Cohen's kappa |\n| Ordinal scale (1-5 rating) | Spearman's rho, Kendall's tau | Cohen's kappa (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision\u002Frecall |\n\nPrioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nBuild direct scoring with three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration** — Choose scale granularity based on rubric detail:\n- 1-3: Binary with neutral option, lowest cognitive load\n- 1-5: Standard Likert, best balance of granularity and reliability\n- 1-10: Use only with detailed per-level rubrics because calibration is harder\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\nAlways require justification before the score in all scoring prompts because research shows this improves reliability by 15-25% compared to score-first approaches.\n\n### Pairwise Comparison Implementation\n\nApply position bias mitigation in every pairwise evaluation:\n\n1. First pass: Response A in first position, Response B in second\n2. Second pass: Response B in first position, Response A in second\n3. Consistency check: If passes disagree, return TIE with reduced confidence\n4. Final verdict: Consistent winner with averaged confidence\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration** — Map confidence to position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nGenerate rubrics to reduce evaluation variance by 40-60% compared to open-ended scoring.\n\n**Include these rubric components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Set strictness calibration** for the use case:\n- **Lenient**: Lower passing bar, appropriate for encouraging iteration\n- **Balanced**: Typical production expectations\n- **Strict**: High standards for safety-critical or high-stakes evaluation\n\nAdapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nBuild production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See [Evaluation Pipeline Diagram](.\u002Freferences\u002Fevaluation-pipeline.md) for the full visual layout.\n\n### Decision Framework: Direct vs. Pairwise\n\nApply this decision tree:\n\n```\nIs there an objective ground truth?\n+-- Yes -> Direct Scoring\n|   Examples: factual accuracy, instruction following, format compliance\n|\n+-- No -> Is it a preference or quality judgment?\n    +-- Yes -> Pairwise Comparison\n    |   Examples: tone, style, persuasiveness, creativity\n    |\n    +-- No -> Consider reference-based evaluation\n        Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation, apply one of these strategies:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.\n\n2. **Hierarchical evaluation**: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.\n\n3. **Human-in-the-loop**: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,\ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n  \"criterion\": \"Factual Accuracy\",\n  \"score\": 5,\n  \"evidence\": [\n    \"Correctly identifies axial tilt as primary cause\",\n    \"Correctly explains differential sunlight by hemisphere\",\n    \"No factual errors present\"\n  ],\n  \"justification\": \"Response accurately explains the cause of seasons with correct\nscientific reasoning. Both the axial tilt and its effect on sunlight distribution\nare correctly described.\",\n  \"improvement\": \"Could add the specific tilt angle (23.5 degrees) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n  \"winner\": \"B\",\n  \"confidence\": 0.7,\n  \"positionConsistency\": {\n    \"consistent\": true,\n    \"firstPassWinner\": \"B\",\n    \"secondPassWinner\": \"B\"\n  }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n  \"levels\": [\n    {\n      \"score\": 1,\n      \"label\": \"Poor\",\n      \"description\": \"Code is difficult to understand without significant effort\",\n      \"characteristics\": [\n        \"No meaningful variable or function names\",\n        \"No comments or documentation\",\n        \"Deeply nested or convoluted logic\"\n      ]\n    },\n    {\n      \"score\": 3,\n      \"label\": \"Adequate\",\n      \"description\": \"Code is understandable with some effort\",\n      \"characteristics\": [\n        \"Most variables have meaningful names\",\n        \"Basic comments present for complex sections\",\n        \"Logic is followable but could be cleaner\"\n      ]\n    },\n    {\n      \"score\": 5,\n      \"label\": \"Excellent\",\n      \"description\": \"Code is immediately clear and maintainable\",\n      \"characteristics\": [\n        \"All names are descriptive and consistent\",\n        \"Comprehensive documentation\",\n        \"Clean, modular structure\"\n      ]\n    }\n  ],\n  \"edgeCases\": [\n    {\n      \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n      \"guidance\": \"Score based on readability for domain experts, not general audience\"\n    }\n  ]\n}\n```\n\n## Guidelines\n\n1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25%\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Gotchas\n\n1. **Scoring without justification**: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.\n\n2. **Single-pass pairwise comparison**: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.\n\n3. **Overloaded criteria**: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.\n\n4. **Missing edge case guidance**: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.\n\n5. **Ignoring confidence calibration**: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.\n\n6. **Rubric drift**: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.\n\n7. **Evaluation prompt sensitivity**: Minor wording changes in evaluation prompts (e.g., reordering instructions, changing phrasing) can cause 10-20% score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.\n\n8. **Uncontrolled length bias**: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.\n\n## Integration\n\nThis skill integrates with:\n\n- **context-fundamentals** - Evaluation prompts require effective context structure\n- **tool-design** - Evaluation tools need proper schemas and error handling\n- **context-optimization** - Evaluation prompts can be optimized for token efficiency\n- **evaluation** (foundational) - This skill extends the foundational evaluation concepts\n\n## References\n\nInternal reference:\n- [LLM-as-Judge Implementation Patterns](.\u002Freferences\u002Fimplementation-patterns.md) - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI\u002FCD\n- [Bias Mitigation Techniques](.\u002Freferences\u002Fbias-mitigation.md) - Read when: evaluation results show inconsistent or suspicious scoring patterns\n- [Metric Selection Guide](.\u002Freferences\u002Fmetrics-guide.md) - Read when: choosing statistical metrics to validate evaluation reliability\n- [Evaluation Pipeline Diagram](.\u002Freferences\u002Fevaluation-pipeline.md) - Read when: designing the architecture of a multi-stage evaluation system\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https:\u002F\u002Feugeneyan.com\u002Fwriting\u002Fllm-evaluators\u002F) - Read when: surveying the state of the art in LLM evaluation\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685) - Read when: understanding position bias and MT-Bench methodology\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634) - Read when: implementing chain-of-thought evaluation scoring\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.17926) - Read when: diagnosing systematic bias in evaluation outputs\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-24\n**Last Updated**: 2026-03-17\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.0.0\n","# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building automated evaluation pipelines for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A\u002FB tests for prompt or model changes\n- Creating rubrics for human or automated evaluation\n- Analyzing correlation between automated and human judgments\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nSelect between two primary approaches based on whether ground truth exists:\n\n**Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.\n\n**Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Achieves higher human-judge agreement than direct scoring for preference tasks (Zheng et al., 2023). Watch for position bias and length bias.\n\n### The Bias Landscape\n\nMitigate these systematic biases in every evaluation system:\n\n**Position Bias**: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.\n\n**Length Bias**: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.\n\n**Verbosity Bias**: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.\n\n### Metric Selection Framework\n\nMatch metrics to the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass\u002Ffail) | Recall, Precision, F1 | Cohen's kappa |\n| Ordinal scale (1-5 rating) | Spearman's rho, Kendall's tau | Cohen's kappa (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision\u002Frecall |\n\nPrioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nBuild direct scoring with three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration** — Choose scale granularity based on rubric detail:\n- 1-3: Binary with neutral option, lowest cognitive load\n- 1-5: Standard Likert, best balance of granularity and reliability\n- 1-10: Use only with detailed per-level rubrics because calibration is harder\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\nAlways require justification before the score in all scoring prompts because research shows this improves reliability by 15-25% compared to score-first approaches.\n\n### Pairwise Comparison Implementation\n\nApply position bias mitigation in every pairwise evaluation:\n\n1. First pass: Response A in first position, Response B in second\n2. Second pass: Response B in first position, Response A in second\n3. Consistency check: If passes disagree, return TIE with reduced confidence\n4. Final verdict: Consistent winner with averaged confidence\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration** — Map confidence to position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nGenerate rubrics to reduce evaluation variance by 40-60% compared to open-ended scoring.\n\n**Include these rubric components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Set strictness calibration** for the use case:\n- **Lenient**: Lower passing bar, appropriate for encouraging iteration\n- **Balanced**: Typical production expectations\n- **Strict**: High standards for safety-critical or high-stakes evaluation\n\nAdapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nBuild production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See [Evaluation Pipeline Diagram](.\u002Freferences\u002Fevaluation-pipeline.md) for the full visual layout.\n\n### Decision Framework: Direct vs. Pairwise\n\nApply this decision tree:\n\n```\nIs there an objective ground truth?\n+-- Yes -> Direct Scoring\n|   Examples: factual accuracy, instruction following, format compliance\n|\n+-- No -> Is it a preference or quality judgment?\n    +-- Yes -> Pairwise Comparison\n    |   Examples: tone, style, persuasiveness, creativity\n    |\n    +-- No -> Consider reference-based evaluation\n        Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation, apply one of these strategies:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.\n\n2. **Hierarchical evaluation**: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.\n\n3. **Human-in-the-loop**: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,\ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n  \"criterion\": \"Factual Accuracy\",\n  \"score\": 5,\n  \"evidence\": [\n    \"Correctly identifies axial tilt as primary cause\",\n    \"Correctly explains differential sunlight by hemisphere\",\n    \"No factual errors present\"\n  ],\n  \"justification\": \"Response accurately explains the cause of seasons with correct\nscientific reasoning. Both the axial tilt and its effect on sunlight distribution\nare correctly described.\",\n  \"improvement\": \"Could add the specific tilt angle (23.5 degrees) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n  \"winner\": \"B\",\n  \"confidence\": 0.7,\n  \"positionConsistency\": {\n    \"consistent\": true,\n    \"firstPassWinner\": \"B\",\n    \"secondPassWinner\": \"B\"\n  }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n  \"levels\": [\n    {\n      \"score\": 1,\n      \"label\": \"Poor\",\n      \"description\": \"Code is difficult to understand without significant effort\",\n      \"characteristics\": [\n        \"No meaningful variable or function names\",\n        \"No comments or documentation\",\n        \"Deeply nested or convoluted logic\"\n      ]\n    },\n    {\n      \"score\": 3,\n      \"label\": \"Adequate\",\n      \"description\": \"Code is understandable with some effort\",\n      \"characteristics\": [\n        \"Most variables have meaningful names\",\n        \"Basic comments present for complex sections\",\n        \"Logic is followable but could be cleaner\"\n      ]\n    },\n    {\n      \"score\": 5,\n      \"label\": \"Excellent\",\n      \"description\": \"Code is immediately clear and maintainable\",\n      \"characteristics\": [\n        \"All names are descriptive and consistent\",\n        \"Comprehensive documentation\",\n        \"Clean, modular structure\"\n      ]\n    }\n  ],\n  \"edgeCases\": [\n    {\n      \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n      \"guidance\": \"Score based on readability for domain experts, not general audience\"\n    }\n  ]\n}\n```\n\n## Guidelines\n\n1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25%\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Gotchas\n\n1. **Scoring without justification**: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.\n\n2. **Single-pass pairwise comparison**: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.\n\n3. **Overloaded criteria**: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.\n\n4. **Missing edge case guidance**: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.\n\n5. **Ignoring confidence calibration**: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.\n\n6. **Rubric drift**: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.\n\n7. **Evaluation prompt sensitivity**: Minor wording changes in evaluation prompts (e.g., reordering instructions, changing phrasing) can cause 10-20% score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.\n\n8. **Uncontrolled length bias**: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.\n\n## Integration\n\nThis skill integrates with:\n\n- **context-fundamentals** - Evaluation prompts require effective context structure\n- **tool-design** - Evaluation tools need proper schemas and error handling\n- **context-optimization** - Evaluation prompts can be optimized for token efficiency\n- **evaluation** (foundational) - This skill extends the foundational evaluation concepts\n\n## References\n\nInternal reference:\n- [LLM-as-Judge Implementation Patterns](.\u002Freferences\u002Fimplementation-patterns.md) - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI\u002FCD\n- [Bias Mitigation Techniques](.\u002Freferences\u002Fbias-mitigation.md) - Read when: evaluation results show inconsistent or suspicious scoring patterns\n- [Metric Selection Guide](.\u002Freferences\u002Fmetrics-guide.md) - Read when: choosing statistical metrics to validate evaluation reliability\n- [Evaluation Pipeline Diagram](.\u002Freferences\u002Fevaluation-pipeline.md) - Read when: designing the architecture of a multi-stage evaluation system\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https:\u002F\u002Feugeneyan.com\u002Fwriting\u002Fllm-evaluators\u002F) - Read when: surveying the state of the art in LLM evaluation\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685) - Read when: understanding position bias and MT-Bench methodology\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634) - Read when: implementing chain-of-thought evaluation scoring\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.17926) - Read when: diagnosing systematic bias in evaluation outputs\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-24\n**Last Updated**: 2026-03-17\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.0.0\n",{"name":5,"description":7},"main","d69096e239f2a89aabc2cd078c47664a9d3b1e11","05-技能与代理开发\u002Fadvanced-evaluation",3,"https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts","https:\u002F\u002Fgithub.com\u002Flza6","个人vx：Tf00798\r\n个人✈：@TFGY999","2026-10-08 14:29:22",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add lza6\u002FArmor-piercing-system-prompts --skill advanced-evaluation",{"tool":38,"command":39},"bunx","bunx skills add lza6\u002FArmor-piercing-system-prompts --skill advanced-evaluation",{"tool":41,"command":42},"pnpm","pnpm dlx skills add lza6\u002FArmor-piercing-system-prompts --skill advanced-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F05-技能与代理开发\u002Fadvanced-evaluation\u002FSKILL.md",true,[46,52,56,60],{"path":47,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":50,"sourceHtmlUrl":51},"references\u002Fbias-mitigation.md","REFERENCE","参考文档",9078,"https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F05-技能与代理开发\u002Fadvanced-evaluation\u002Freferences\u002Fbias-mitigation.md",{"path":53,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002Fevaluation-pipeline.md",2891,"https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F05-技能与代理开发\u002Fadvanced-evaluation\u002Freferences\u002Fevaluation-pipeline.md",{"path":57,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fimplementation-patterns.md",9061,"https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F05-技能与代理开发\u002Fadvanced-evaluation\u002Freferences\u002Fimplementation-patterns.md",{"path":61,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fmetrics-guide.md",9333,"https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F05-技能与代理开发\u002Fadvanced-evaluation\u002Freferences\u002Fmetrics-guide.md",[65,73,78,85,92,99],{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":72},"01M7FFG485KY83TD6HSC4YPCMC","obsidian-clipper-template-creator","Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F07-后端与架构\u002Fobsidian-clipper-template-creator\u002FSKILL.md",[],[],"2026-11-16 10:05:34",{"skillNo":74,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":72},"01M7FFG0T6AZ0STEJPSB3CX84K","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F07-后端与架构\u002Fobsidian-clipper-template-creator-antigravity-awesome-skills-main\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":72},"01M7FFFY0QEM14SKWPDYGMXM2P","python-api","Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP\u002FmAP semantics, and source-style import diagnostics.","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F07-后端与架构\u002Fobject-detection-metrics\u002Fsub-skills\u002Fpython-api\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":72},"01M7FFFVZQZ2GJ0C0TGP7JHD20","file-evaluation","Folder-based PASCAL VOC AP\u002FmAP evaluation from ground-truth and detection text files.","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F07-后端与架构\u002Fobject-detection-metrics\u002Fsub-skills\u002Ffile-evaluation\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":72},"01M7FFFSFNYSE77SJ6PE1CK2DA","object-detection-metrics","Guides legacy Object-Detection-Metrics PASCAL VOC AP\u002FmAP evaluation from detection text folders and direct Python API classes.","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F07-后端与架构\u002Fobject-detection-metrics\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":72},"01M7FFFP0GCK1RVXBCPQ8APBMT","nx-workspace-patterns","Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.","https:\u002F\u002Fgithub.com\u002Flza6\u002FArmor-piercing-system-prompts\u002Fblob\u002Fmain\u002F07-后端与架构\u002Fnx-workspace-patterns\u002FSKILL.md",[],[]]