[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7152Q07EH3P9MGWM37EGD7B":3,"skill-same:01M7152Q07EH3P9MGWM37EGD7B":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M7152Q07EH3P9MGWM37EGD7B","advanced-evaluation",null,"This skill should be used when the user asks to \"implement LLM-as-judge\", \"compare model outputs\", \"create evaluation rubrics\", \"mitigate evaluation bias\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.","sapsapshen\u002Fai-memory-router","sapsapshen","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F20347959?v=4","01M6XDQEF1DP50DKF37H81J3J5","shenmian","01M6XDQEJPE41Y3DT2763X8ZJN",0,"MIT","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fcursor\u002Fadvanced-evaluation\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"Evaluation","advanced evaluation","advanced","bias","Quality Assessment","Pairwise Comparison","LLM-as-Judge","2026","when","evaluation rubrics","judge","as","safe","scoring","wed","community","direct","model","create","assessment","2026-11-10 10:12:30",false,"---\nname: advanced-evaluation\ndescription: This skill should be used when the user asks to \"implement LLM-as-judge\", \"compare model outputs\", \"create evaluation rubrics\", \"mitigate evaluation bias\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.\nrisk: safe\nsource: community\ndate_added: 2026-03-18\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Use\nActivate this skill when:\n\n- Building automated evaluation pipelines for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A\u002FB tests for prompt or model changes\n- Creating rubrics for human or automated evaluation\n- Analyzing correlation between automated and human judgments\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nEvaluation approaches fall into two primary categories with distinct reliability profiles:\n\n**Direct Scoring**: A single LLM rates one response on a defined scale.\n- Best for: Objective criteria (factual accuracy, instruction following, toxicity)\n- Reliability: Moderate to high for well-defined criteria\n- Failure mode: Score calibration drift, inconsistent scale interpretation\n\n**Pairwise Comparison**: An LLM compares two responses and selects the better one.\n- Best for: Subjective preferences (tone, style, persuasiveness)\n- Reliability: Higher than direct scoring for preferences\n- Failure mode: Position bias, length bias\n\nResearch from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.\n\n### The Bias Landscape\n\nLLM judges exhibit systematic biases that must be actively mitigated:\n\n**Position Bias**: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.\n\n**Length Bias**: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.\n\n**Verbosity Bias**: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer.\n\n### Metric Selection Framework\n\nChoose metrics based on the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass\u002Ffail) | Recall, Precision, F1 | Cohen's κ |\n| Ordinal scale (1-5 rating) | Spearman's ρ, Kendall's τ | Cohen's κ (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision\u002Frecall |\n\nThe critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nDirect scoring requires three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration**:\n- 1-3 scales: Binary with neutral option, lowest cognitive load\n- 1-5 scales: Standard Likert, good balance of granularity and reliability\n- 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\n**Chain-of-Thought Requirement**: All scoring prompts must require justification before the score. Research shows this improves reliability by 15-25% compared to score-first approaches.\n\n### Pairwise Comparison Implementation\n\nPairwise comparison is inherently more reliable for preference-based evaluation but requires bias mitigation.\n\n**Position Bias Mitigation Protocol**:\n1. First pass: Response A in first position, Response B in second\n2. Second pass: Response B in first position, Response A in second\n3. Consistency check: If passes disagree, return TIE with reduced confidence\n4. Final verdict: Consistent winner with averaged confidence\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration**: Confidence scores should reflect position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nWell-defined rubrics reduce evaluation variance by 40-60% compared to open-ended scoring.\n\n**Rubric Components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Strictness Calibration**:\n- **Lenient**: Lower bar for passing scores, appropriate for encouraging iteration\n- **Balanced**: Fair, typical expectations for production use\n- **Strict**: High standards, appropriate for safety-critical or high-stakes evaluation\n\n**Domain Adaptation**: Rubrics should use domain-specific terminology. A \"code readability\" rubric mentions variables, functions, and comments. A \"medical accuracy\" rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nProduction evaluation systems require multiple layers:\n\n```\n┌─────────────────────────────────────────────────┐\n│                 Evaluation Pipeline              │\n├─────────────────────────────────────────────────┤\n│                                                   │\n│  Input: Response + Prompt + Context               │\n│           │                                       │\n│           ▼                                       │\n│  ┌─────────────────────┐                         │\n│  │   Criteria Loader   │ ◄── Rubrics, weights    │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Primary Scorer    │ ◄── Direct or Pairwise  │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Bias Mitigation   │ ◄── Position swap, etc. │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │ Confidence Scoring  │ ◄── Calibration         │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  Output: Scores + Justifications + Confidence     │\n│                                                   │\n└─────────────────────────────────────────────────┘\n```\n\n### Common Anti-Patterns\n\n**Anti-pattern: Scoring without justification**\n- Problem: Scores lack grounding, difficult to debug or improve\n- Solution: Always require evidence-based justification before score\n\n**Anti-pattern: Single-pass pairwise comparison**\n- Problem: Position bias corrupts results\n- Solution: Always swap positions and check consistency\n\n**Anti-pattern: Overloaded criteria**\n- Problem: Criteria measuring multiple things are unreliable\n- Solution: One criterion = one measurable aspect\n\n**Anti-pattern: Missing edge case guidance**\n- Problem: Evaluators handle ambiguous cases inconsistently\n- Solution: Include edge cases in rubrics with explicit guidance\n\n**Anti-pattern: Ignoring confidence calibration**\n- Problem: High-confidence wrong judgments are worse than low-confidence\n- Solution: Calibrate confidence to position consistency and evidence strength\n\n### Decision Framework: Direct vs. Pairwise\n\nUse this decision tree:\n\n```\nIs there an objective ground truth?\n├── Yes → Direct Scoring\n│   └── Examples: factual accuracy, instruction following, format compliance\n│\n└── No → Is it a preference or quality judgment?\n    ├── Yes → Pairwise Comparison\n    │   └── Examples: tone, style, persuasiveness, creativity\n    │\n    └── No → Consider reference-based evaluation\n        └── Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges, aggregate votes\n   - Reduces individual model bias\n   - More expensive but more reliable for high-stakes decisions\n\n2. **Hierarchical evaluation**: Fast cheap model for screening, expensive model for edge cases\n   - Cost-effective for large volumes\n   - Requires calibration of screening threshold\n\n3. **Human-in-the-loop**: Automated evaluation for clear cases, human review for low-confidence\n   - Best reliability for critical applications\n   - Design feedback loop to improve automated evaluation\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun, \ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n  \"criterion\": \"Factual Accuracy\",\n  \"score\": 5,\n  \"evidence\": [\n    \"Correctly identifies axial tilt as primary cause\",\n    \"Correctly explains differential sunlight by hemisphere\",\n    \"No factual errors present\"\n  ],\n  \"justification\": \"Response accurately explains the cause of seasons with correct \nscientific reasoning. Both the axial tilt and its effect on sunlight distribution \nare correctly described.\",\n  \"improvement\": \"Could add the specific tilt angle (23.5°) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n  \"winner\": \"B\",\n  \"confidence\": 0.7,\n  \"positionConsistency\": {\n    \"consistent\": true,\n    \"firstPassWinner\": \"B\",\n    \"secondPassWinner\": \"B\"\n  }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n  \"levels\": [\n    {\n      \"score\": 1,\n      \"label\": \"Poor\",\n      \"description\": \"Code is difficult to understand without significant effort\",\n      \"characteristics\": [\n        \"No meaningful variable or function names\",\n        \"No comments or documentation\",\n        \"Deeply nested or convoluted logic\"\n      ]\n    },\n    {\n      \"score\": 3,\n      \"label\": \"Adequate\",\n      \"description\": \"Code is understandable with some effort\",\n      \"characteristics\": [\n        \"Most variables have meaningful names\",\n        \"Basic comments present for complex sections\",\n        \"Logic is followable but could be cleaner\"\n      ]\n    },\n    {\n      \"score\": 5,\n      \"label\": \"Excellent\",\n      \"description\": \"Code is immediately clear and maintainable\",\n      \"characteristics\": [\n        \"All names are descriptive and consistent\",\n        \"Comprehensive documentation\",\n        \"Clean, modular structure\"\n      ]\n    }\n  ],\n  \"edgeCases\": [\n    {\n      \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n      \"guidance\": \"Score based on readability for domain experts, not general audience\"\n    }\n  ]\n}\n```\n\n## Guidelines\n\n1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25%\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Integration\n\nThis skill integrates with:\n\n- **context-fundamentals** - Evaluation prompts require effective context structure\n- **tool-design** - Evaluation tools need proper schemas and error handling\n- **context-optimization** - Evaluation prompts can be optimized for token efficiency\n- **evaluation** (foundational) - This skill extends the foundational evaluation concepts\n\n## References\n\nInternal reference:\n- LLM-as-Judge Implementation Patterns\n- Bias Mitigation Techniques\n- Metric Selection Guide\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https:\u002F\u002Feugeneyan.com\u002Fwriting\u002Fllm-evaluators\u002F)\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685)\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634)\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.17926)\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-24\n**Last Updated**: 2024-12-24\n**Author**: Muratcan Koylan\n**Version**: 1.0.0\n\n","# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Use\nActivate this skill when:\n\n- Building automated evaluation pipelines for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A\u002FB tests for prompt or model changes\n- Creating rubrics for human or automated evaluation\n- Analyzing correlation between automated and human judgments\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nEvaluation approaches fall into two primary categories with distinct reliability profiles:\n\n**Direct Scoring**: A single LLM rates one response on a defined scale.\n- Best for: Objective criteria (factual accuracy, instruction following, toxicity)\n- Reliability: Moderate to high for well-defined criteria\n- Failure mode: Score calibration drift, inconsistent scale interpretation\n\n**Pairwise Comparison**: An LLM compares two responses and selects the better one.\n- Best for: Subjective preferences (tone, style, persuasiveness)\n- Reliability: Higher than direct scoring for preferences\n- Failure mode: Position bias, length bias\n\nResearch from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.\n\n### The Bias Landscape\n\nLLM judges exhibit systematic biases that must be actively mitigated:\n\n**Position Bias**: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.\n\n**Length Bias**: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.\n\n**Verbosity Bias**: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer.\n\n### Metric Selection Framework\n\nChoose metrics based on the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass\u002Ffail) | Recall, Precision, F1 | Cohen's κ |\n| Ordinal scale (1-5 rating) | Spearman's ρ, Kendall's τ | Cohen's κ (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision\u002Frecall |\n\nThe critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nDirect scoring requires three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration**:\n- 1-3 scales: Binary with neutral option, lowest cognitive load\n- 1-5 scales: Standard Likert, good balance of granularity and reliability\n- 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\n**Chain-of-Thought Requirement**: All scoring prompts must require justification before the score. Research shows this improves reliability by 15-25% compared to score-first approaches.\n\n### Pairwise Comparison Implementation\n\nPairwise comparison is inherently more reliable for preference-based evaluation but requires bias mitigation.\n\n**Position Bias Mitigation Protocol**:\n1. First pass: Response A in first position, Response B in second\n2. Second pass: Response B in first position, Response A in second\n3. Consistency check: If passes disagree, return TIE with reduced confidence\n4. Final verdict: Consistent winner with averaged confidence\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration**: Confidence scores should reflect position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nWell-defined rubrics reduce evaluation variance by 40-60% compared to open-ended scoring.\n\n**Rubric Components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Strictness Calibration**:\n- **Lenient**: Lower bar for passing scores, appropriate for encouraging iteration\n- **Balanced**: Fair, typical expectations for production use\n- **Strict**: High standards, appropriate for safety-critical or high-stakes evaluation\n\n**Domain Adaptation**: Rubrics should use domain-specific terminology. A \"code readability\" rubric mentions variables, functions, and comments. A \"medical accuracy\" rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nProduction evaluation systems require multiple layers:\n\n```\n┌─────────────────────────────────────────────────┐\n│                 Evaluation Pipeline              │\n├─────────────────────────────────────────────────┤\n│                                                   │\n│  Input: Response + Prompt + Context               │\n│           │                                       │\n│           ▼                                       │\n│  ┌─────────────────────┐                         │\n│  │   Criteria Loader   │ ◄── Rubrics, weights    │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Primary Scorer    │ ◄── Direct or Pairwise  │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Bias Mitigation   │ ◄── Position swap, etc. │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │ Confidence Scoring  │ ◄── Calibration         │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  Output: Scores + Justifications + Confidence     │\n│                                                   │\n└─────────────────────────────────────────────────┘\n```\n\n### Common Anti-Patterns\n\n**Anti-pattern: Scoring without justification**\n- Problem: Scores lack grounding, difficult to debug or improve\n- Solution: Always require evidence-based justification before score\n\n**Anti-pattern: Single-pass pairwise comparison**\n- Problem: Position bias corrupts results\n- Solution: Always swap positions and check consistency\n\n**Anti-pattern: Overloaded criteria**\n- Problem: Criteria measuring multiple things are unreliable\n- Solution: One criterion = one measurable aspect\n\n**Anti-pattern: Missing edge case guidance**\n- Problem: Evaluators handle ambiguous cases inconsistently\n- Solution: Include edge cases in rubrics with explicit guidance\n\n**Anti-pattern: Ignoring confidence calibration**\n- Problem: High-confidence wrong judgments are worse than low-confidence\n- Solution: Calibrate confidence to position consistency and evidence strength\n\n### Decision Framework: Direct vs. Pairwise\n\nUse this decision tree:\n\n```\nIs there an objective ground truth?\n├── Yes → Direct Scoring\n│   └── Examples: factual accuracy, instruction following, format compliance\n│\n└── No → Is it a preference or quality judgment?\n    ├── Yes → Pairwise Comparison\n    │   └── Examples: tone, style, persuasiveness, creativity\n    │\n    └── No → Consider reference-based evaluation\n        └── Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges, aggregate votes\n   - Reduces individual model bias\n   - More expensive but more reliable for high-stakes decisions\n\n2. **Hierarchical evaluation**: Fast cheap model for screening, expensive model for edge cases\n   - Cost-effective for large volumes\n   - Requires calibration of screening threshold\n\n3. **Human-in-the-loop**: Automated evaluation for clear cases, human review for low-confidence\n   - Best reliability for critical applications\n   - Design feedback loop to improve automated evaluation\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun, \ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n  \"criterion\": \"Factual Accuracy\",\n  \"score\": 5,\n  \"evidence\": [\n    \"Correctly identifies axial tilt as primary cause\",\n    \"Correctly explains differential sunlight by hemisphere\",\n    \"No factual errors present\"\n  ],\n  \"justification\": \"Response accurately explains the cause of seasons with correct \nscientific reasoning. Both the axial tilt and its effect on sunlight distribution \nare correctly described.\",\n  \"improvement\": \"Could add the specific tilt angle (23.5°) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n  \"winner\": \"B\",\n  \"confidence\": 0.7,\n  \"positionConsistency\": {\n    \"consistent\": true,\n    \"firstPassWinner\": \"B\",\n    \"secondPassWinner\": \"B\"\n  }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n  \"levels\": [\n    {\n      \"score\": 1,\n      \"label\": \"Poor\",\n      \"description\": \"Code is difficult to understand without significant effort\",\n      \"characteristics\": [\n        \"No meaningful variable or function names\",\n        \"No comments or documentation\",\n        \"Deeply nested or convoluted logic\"\n      ]\n    },\n    {\n      \"score\": 3,\n      \"label\": \"Adequate\",\n      \"description\": \"Code is understandable with some effort\",\n      \"characteristics\": [\n        \"Most variables have meaningful names\",\n        \"Basic comments present for complex sections\",\n        \"Logic is followable but could be cleaner\"\n      ]\n    },\n    {\n      \"score\": 5,\n      \"label\": \"Excellent\",\n      \"description\": \"Code is immediately clear and maintainable\",\n      \"characteristics\": [\n        \"All names are descriptive and consistent\",\n        \"Comprehensive documentation\",\n        \"Clean, modular structure\"\n      ]\n    }\n  ],\n  \"edgeCases\": [\n    {\n      \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n      \"guidance\": \"Score based on readability for domain experts, not general audience\"\n    }\n  ]\n}\n```\n\n## Guidelines\n\n1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25%\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Integration\n\nThis skill integrates with:\n\n- **context-fundamentals** - Evaluation prompts require effective context structure\n- **tool-design** - Evaluation tools need proper schemas and error handling\n- **context-optimization** - Evaluation prompts can be optimized for token efficiency\n- **evaluation** (foundational) - This skill extends the foundational evaluation concepts\n\n## References\n\nInternal reference:\n- LLM-as-Judge Implementation Patterns\n- Bias Mitigation Techniques\n- Metric Selection Guide\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https:\u002F\u002Feugeneyan.com\u002Fwriting\u002Fllm-evaluators\u002F)\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685)\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634)\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.17926)\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-24\n**Last Updated**: 2024-12-24\n**Author**: Muratcan Koylan\n**Version**: 1.0.0\n\n",{"name":5,"risk":31,"source":34,"date_added":44,"description":7},1773792000000,"main","7a38d8e7073e61c16eda85681cb32fb68a8657a5","skills-store\u002Fcursor\u002Fadvanced-evaluation","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router","https:\u002F\u002Fgithub.com\u002Fsapsapshen","2026-04-14 21:12:12",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add sapsapshen\u002Fai-memory-router --skill advanced-evaluation",{"tool":57,"command":58},"bunx","bunx skills add sapsapshen\u002Fai-memory-router --skill advanced-evaluation",{"tool":60,"command":61},"pnpm","pnpm dlx skills add sapsapshen\u002Fai-memory-router --skill advanced-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fcursor\u002Fadvanced-evaluation\u002FSKILL.md",true,[],[66,90,115,125,138,157],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":77,"publishedAt":39},"01M71ATQRKG48SH06FQ697T5G9","wechat-search","Search WeChat Official Account articles using OpenClaw's web search and fetch capabilities with compliance-focused design.","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Fwechat-search-release\u002FSKILL.md",[72],{"categoryNo":73,"categoryName":74,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000037","律师与法务","PROFESSION","适用职业",[78,79,80,81,82,83,84,85,86,87,88,89],"wechat search","wechat","search","capabilities","openclaw","design","web search","compliance","web","official","fetch","with",{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":39},"01M71ATNAVGZB35PFYM2GC5N00","trend-tap","Real-time trending topics aggregator across 7 platforms (X\u002FTwitter, Reddit, Google Trends, Hacker News, Zhihu, Bilibili, Weibo). Trigger: when user says 'trend', 'trends', 'trend tap', '热点', '热搜', or asks what is trending\u002Fhot. Do NOT use web_search — this skill already fetches live data from all platforms with zero API keys.","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Ftrend-tap\u002FSKILL.md",[],[97,98,99,100,101,102,80,103,104,105,106,27,107,108,109,110,111,112,113,114],"trend","trend tap","tap","api keys","Bash","time","trending","hacker","aggregator","do","brew","live","data","google","install","read","across","api",{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":39},"01M71ATJ9AM5KM15AA8PS302EB","trend-radar","Real-time trending topics aggregator across 7 platforms (X\u002FTwitter, Reddit, Google Trends, Hacker News, Zhihu, Bilibili, Weibo). Trigger: when user says 'trend', 'trends', 'trend pulse', '热点', '热搜', or asks what is trending\u002Fhot. Do NOT use web_search — this skill already fetches live data from all platforms with zero API keys.","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Ftrend-radar\u002FSKILL.md",[],[122,123,97,107,108,109,110,101,80,112,124,113,114,27,105,104,103,111,102,100],"trend radar","radar","pulse",{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":131,"publishedAt":39},"01M71ATGF095N3EJM9G8SD0BB5","tech-news-daily","每日科技新闻汇——从全球多个 RSS 源抓取最新科技新闻，翻译（英→中）、去重、排版成每日简报。当用户询问科技新闻、科技新鲜事、每日科技、技术动态、Daily Tech News 或类似表达时触发此技能。","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Ftech-news-daily\u002FSKILL.md",[],[132,133,134,135,136,137],"news","tech","tech news daily","daily","rss","翻译",{"skillNo":139,"name":140,"nameZh":6,"description":141,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":142,"categories":143,"keywords":149,"publishedAt":39},"01M71ATFAEAMQ17M70NQNY7SR8","summarize","Summarize URLs or files with the summarize CLI (web, PDFs, images, audio, YouTube).","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Fsummarize\u002FSKILL.md",[144],{"categoryNo":145,"categoryName":146,"taxonomyType":147,"taxonomyTypeDesc":148},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[140,86,150,151,152,89,107,153,154,155,99,156,111],"Audio","cli","youtube","images","https","files","pdfs",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":163,"publishedAt":39},"01M71ATDNS3R0M4H936PACQAWZ","sonoscli","Control Sonos speakers (discover\u002Fstatus\u002Fplay\u002Fvolume\u002Fgroup).","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Fsonoscli\u002FSKILL.md",[],[159,164,165,166,111,167,154,168,169,170,171,172],"status","play","go","latest","github","discover","control","cmd","volume"]