[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M736NG1RTZ5GEV3FC2X5F2WF":3,"skill-same:01M736NG1RTZ5GEV3FC2X5F2WF":46},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":14,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":29,"ownerDescription":30,"updatedAt":31,"install":32,"resources":45},"01M736NG1RTZ5GEV3FC2X5F2WF","advanced-evaluation",null,"This skill should be used when the user asks to \"implement LLM-as-judge\", \"compare model outputs\", \"create evaluation rubrics\", \"mitigate evaluation bias\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.","kalash33\u002Fkstack","kalash33","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F90259149?v=4","01M7365X8Q0B73NYHA730SMZWN","Kalash Poddar","01M7365X9D0JK47C3Q0YCCK9Z2",0,"MIT","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fadvanced-evaluation\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: advanced-evaluation\ndescription: This skill should be used when the user asks to \"implement LLM-as-judge\", \"compare model outputs\", \"create evaluation rubrics\", \"mitigate evaluation bias\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building automated evaluation pipelines for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A\u002FB tests for prompt or model changes\n- Creating rubrics for human or automated evaluation\n- Analyzing correlation between automated and human judgments\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nEvaluation approaches fall into two primary categories with distinct reliability profiles:\n\n**Direct Scoring**: A single LLM rates one response on a defined scale.\n- Best for: Objective criteria (factual accuracy, instruction following, toxicity)\n- Reliability: Moderate to high for well-defined criteria\n- Failure mode: Score calibration drift, inconsistent scale interpretation\n\n**Pairwise Comparison**: An LLM compares two responses and selects the better one.\n- Best for: Subjective preferences (tone, style, persuasiveness)\n- Reliability: Higher than direct scoring for preferences\n- Failure mode: Position bias, length bias\n\nResearch from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.\n\n### The Bias Landscape\n\nLLM judges exhibit systematic biases that must be actively mitigated:\n\n**Position Bias**: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.\n\n**Length Bias**: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.\n\n**Verbosity Bias**: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer.\n\n### Metric Selection Framework\n\nChoose metrics based on the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass\u002Ffail) | Recall, Precision, F1 | Cohen's κ |\n| Ordinal scale (1-5 rating) | Spearman's ρ, Kendall's τ | Cohen's κ (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision\u002Frecall |\n\nThe critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nDirect scoring requires three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration**:\n- 1-3 scales: Binary with neutral option, lowest cognitive load\n- 1-5 scales: Standard Likert, good balance of granularity and reliability\n- 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\n**Chain-of-Thought Requirement**: All scoring prompts must require justification before the score. Research shows this improves reliability by 15-25% compared to score-first approaches.\n\n### Pairwise Comparison Implementation\n\nPairwise comparison is inherently more reliable for preference-based evaluation but requires bias mitigation.\n\n**Position Bias Mitigation Protocol**:\n1. First pass: Response A in first position, Response B in second\n2. Second pass: Response B in first position, Response A in second\n3. Consistency check: If passes disagree, return TIE with reduced confidence\n4. Final verdict: Consistent winner with averaged confidence\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration**: Confidence scores should reflect position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nWell-defined rubrics reduce evaluation variance by 40-60% compared to open-ended scoring.\n\n**Rubric Components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Strictness Calibration**:\n- **Lenient**: Lower bar for passing scores, appropriate for encouraging iteration\n- **Balanced**: Fair, typical expectations for production use\n- **Strict**: High standards, appropriate for safety-critical or high-stakes evaluation\n\n**Domain Adaptation**: Rubrics should use domain-specific terminology. A \"code readability\" rubric mentions variables, functions, and comments. A \"medical accuracy\" rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nProduction evaluation systems require multiple layers:\n\n```\n┌─────────────────────────────────────────────────┐\n│                 Evaluation Pipeline              │\n├─────────────────────────────────────────────────┤\n│                                                   │\n│  Input: Response + Prompt + Context               │\n│           │                                       │\n│           ▼                                       │\n│  ┌─────────────────────┐                         │\n│  │   Criteria Loader   │ ◄── Rubrics, weights    │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Primary Scorer    │ ◄── Direct or Pairwise  │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Bias Mitigation   │ ◄── Position swap, etc. │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │ Confidence Scoring  │ ◄── Calibration         │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  Output: Scores + Justifications + Confidence     │\n│                                                   │\n└─────────────────────────────────────────────────┘\n```\n\n### Common Anti-Patterns\n\n**Anti-pattern: Scoring without justification**\n- Problem: Scores lack grounding, difficult to debug or improve\n- Solution: Always require evidence-based justification before score\n\n**Anti-pattern: Single-pass pairwise comparison**\n- Problem: Position bias corrupts results\n- Solution: Always swap positions and check consistency\n\n**Anti-pattern: Overloaded criteria**\n- Problem: Criteria measuring multiple things are unreliable\n- Solution: One criterion = one measurable aspect\n\n**Anti-pattern: Missing edge case guidance**\n- Problem: Evaluators handle ambiguous cases inconsistently\n- Solution: Include edge cases in rubrics with explicit guidance\n\n**Anti-pattern: Ignoring confidence calibration**\n- Problem: High-confidence wrong judgments are worse than low-confidence\n- Solution: Calibrate confidence to position consistency and evidence strength\n\n### Decision Framework: Direct vs. Pairwise\n\nUse this decision tree:\n\n```\nIs there an objective ground truth?\n├── Yes → Direct Scoring\n│   └── Examples: factual accuracy, instruction following, format compliance\n│\n└── No → Is it a preference or quality judgment?\n    ├── Yes → Pairwise Comparison\n    │   └── Examples: tone, style, persuasiveness, creativity\n    │\n    └── No → Consider reference-based evaluation\n        └── Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges, aggregate votes\n   - Reduces individual model bias\n   - More expensive but more reliable for high-stakes decisions\n\n2. **Hierarchical evaluation**: Fast cheap model for screening, expensive model for edge cases\n   - Cost-effective for large volumes\n   - Requires calibration of screening threshold\n\n3. **Human-in-the-loop**: Automated evaluation for clear cases, human review for low-confidence\n   - Best reliability for critical applications\n   - Design feedback loop to improve automated evaluation\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun, \ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n  \"criterion\": \"Factual Accuracy\",\n  \"score\": 5,\n  \"evidence\": [\n    \"Correctly identifies axial tilt as primary cause\",\n    \"Correctly explains differential sunlight by hemisphere\",\n    \"No factual errors present\"\n  ],\n  \"justification\": \"Response accurately explains the cause of seasons with correct \nscientific reasoning. Both the axial tilt and its effect on sunlight distribution \nare correctly described.\",\n  \"improvement\": \"Could add the specific tilt angle (23.5°) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n  \"winner\": \"B\",\n  \"confidence\": 0.7,\n  \"positionConsistency\": {\n    \"consistent\": true,\n    \"firstPassWinner\": \"B\",\n    \"secondPassWinner\": \"B\"\n  }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n  \"levels\": [\n    {\n      \"score\": 1,\n      \"label\": \"Poor\",\n      \"description\": \"Code is difficult to understand without significant effort\",\n      \"characteristics\": [\n        \"No meaningful variable or function names\",\n        \"No comments or documentation\",\n        \"Deeply nested or convoluted logic\"\n      ]\n    },\n    {\n      \"score\": 3,\n      \"label\": \"Adequate\",\n      \"description\": \"Code is understandable with some effort\",\n      \"characteristics\": [\n        \"Most variables have meaningful names\",\n        \"Basic comments present for complex sections\",\n        \"Logic is followable but could be cleaner\"\n      ]\n    },\n    {\n      \"score\": 5,\n      \"label\": \"Excellent\",\n      \"description\": \"Code is immediately clear and maintainable\",\n      \"characteristics\": [\n        \"All names are descriptive and consistent\",\n        \"Comprehensive documentation\",\n        \"Clean, modular structure\"\n      ]\n    }\n  ],\n  \"edgeCases\": [\n    {\n      \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n      \"guidance\": \"Score based on readability for domain experts, not general audience\"\n    }\n  ]\n}\n```\n\n## Guidelines\n\n1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25%\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Integration\n\nThis skill integrates with:\n\n- **context-fundamentals** - Evaluation prompts require effective context structure\n- **tool-design** - Evaluation tools need proper schemas and error handling\n- **context-optimization** - Evaluation prompts can be optimized for token efficiency\n- **evaluation** (foundational) - This skill extends the foundational evaluation concepts\n\n## References\n\nInternal reference:\n- LLM-as-Judge Implementation Patterns\n- Bias Mitigation Techniques\n- Metric Selection Guide\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https:\u002F\u002Feugeneyan.com\u002Fwriting\u002Fllm-evaluators\u002F)\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685)\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634)\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.17926)\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-24\n**Last Updated**: 2024-12-24\n**Author**: Muratcan Koylan\n**Version**: 1.0.0\n\n","# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building automated evaluation pipelines for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A\u002FB tests for prompt or model changes\n- Creating rubrics for human or automated evaluation\n- Analyzing correlation between automated and human judgments\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nEvaluation approaches fall into two primary categories with distinct reliability profiles:\n\n**Direct Scoring**: A single LLM rates one response on a defined scale.\n- Best for: Objective criteria (factual accuracy, instruction following, toxicity)\n- Reliability: Moderate to high for well-defined criteria\n- Failure mode: Score calibration drift, inconsistent scale interpretation\n\n**Pairwise Comparison**: An LLM compares two responses and selects the better one.\n- Best for: Subjective preferences (tone, style, persuasiveness)\n- Reliability: Higher than direct scoring for preferences\n- Failure mode: Position bias, length bias\n\nResearch from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.\n\n### The Bias Landscape\n\nLLM judges exhibit systematic biases that must be actively mitigated:\n\n**Position Bias**: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.\n\n**Length Bias**: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.\n\n**Verbosity Bias**: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer.\n\n### Metric Selection Framework\n\nChoose metrics based on the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass\u002Ffail) | Recall, Precision, F1 | Cohen's κ |\n| Ordinal scale (1-5 rating) | Spearman's ρ, Kendall's τ | Cohen's κ (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision\u002Frecall |\n\nThe critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nDirect scoring requires three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration**:\n- 1-3 scales: Binary with neutral option, lowest cognitive load\n- 1-5 scales: Standard Likert, good balance of granularity and reliability\n- 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\n**Chain-of-Thought Requirement**: All scoring prompts must require justification before the score. Research shows this improves reliability by 15-25% compared to score-first approaches.\n\n### Pairwise Comparison Implementation\n\nPairwise comparison is inherently more reliable for preference-based evaluation but requires bias mitigation.\n\n**Position Bias Mitigation Protocol**:\n1. First pass: Response A in first position, Response B in second\n2. Second pass: Response B in first position, Response A in second\n3. Consistency check: If passes disagree, return TIE with reduced confidence\n4. Final verdict: Consistent winner with averaged confidence\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration**: Confidence scores should reflect position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nWell-defined rubrics reduce evaluation variance by 40-60% compared to open-ended scoring.\n\n**Rubric Components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Strictness Calibration**:\n- **Lenient**: Lower bar for passing scores, appropriate for encouraging iteration\n- **Balanced**: Fair, typical expectations for production use\n- **Strict**: High standards, appropriate for safety-critical or high-stakes evaluation\n\n**Domain Adaptation**: Rubrics should use domain-specific terminology. A \"code readability\" rubric mentions variables, functions, and comments. A \"medical accuracy\" rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nProduction evaluation systems require multiple layers:\n\n```\n┌─────────────────────────────────────────────────┐\n│                 Evaluation Pipeline              │\n├─────────────────────────────────────────────────┤\n│                                                   │\n│  Input: Response + Prompt + Context               │\n│           │                                       │\n│           ▼                                       │\n│  ┌─────────────────────┐                         │\n│  │   Criteria Loader   │ ◄── Rubrics, weights    │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Primary Scorer    │ ◄── Direct or Pairwise  │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │   Bias Mitigation   │ ◄── Position swap, etc. │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  ┌─────────────────────┐                         │\n│  │ Confidence Scoring  │ ◄── Calibration         │\n│  └──────────┬──────────┘                         │\n│             │                                     │\n│             ▼                                     │\n│  Output: Scores + Justifications + Confidence     │\n│                                                   │\n└─────────────────────────────────────────────────┘\n```\n\n### Common Anti-Patterns\n\n**Anti-pattern: Scoring without justification**\n- Problem: Scores lack grounding, difficult to debug or improve\n- Solution: Always require evidence-based justification before score\n\n**Anti-pattern: Single-pass pairwise comparison**\n- Problem: Position bias corrupts results\n- Solution: Always swap positions and check consistency\n\n**Anti-pattern: Overloaded criteria**\n- Problem: Criteria measuring multiple things are unreliable\n- Solution: One criterion = one measurable aspect\n\n**Anti-pattern: Missing edge case guidance**\n- Problem: Evaluators handle ambiguous cases inconsistently\n- Solution: Include edge cases in rubrics with explicit guidance\n\n**Anti-pattern: Ignoring confidence calibration**\n- Problem: High-confidence wrong judgments are worse than low-confidence\n- Solution: Calibrate confidence to position consistency and evidence strength\n\n### Decision Framework: Direct vs. Pairwise\n\nUse this decision tree:\n\n```\nIs there an objective ground truth?\n├── Yes → Direct Scoring\n│   └── Examples: factual accuracy, instruction following, format compliance\n│\n└── No → Is it a preference or quality judgment?\n    ├── Yes → Pairwise Comparison\n    │   └── Examples: tone, style, persuasiveness, creativity\n    │\n    └── No → Consider reference-based evaluation\n        └── Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges, aggregate votes\n   - Reduces individual model bias\n   - More expensive but more reliable for high-stakes decisions\n\n2. **Hierarchical evaluation**: Fast cheap model for screening, expensive model for edge cases\n   - Cost-effective for large volumes\n   - Requires calibration of screening threshold\n\n3. **Human-in-the-loop**: Automated evaluation for clear cases, human review for low-confidence\n   - Best reliability for critical applications\n   - Design feedback loop to improve automated evaluation\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun, \ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n  \"criterion\": \"Factual Accuracy\",\n  \"score\": 5,\n  \"evidence\": [\n    \"Correctly identifies axial tilt as primary cause\",\n    \"Correctly explains differential sunlight by hemisphere\",\n    \"No factual errors present\"\n  ],\n  \"justification\": \"Response accurately explains the cause of seasons with correct \nscientific reasoning. Both the axial tilt and its effect on sunlight distribution \nare correctly described.\",\n  \"improvement\": \"Could add the specific tilt angle (23.5°) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n  \"winner\": \"B\",\n  \"confidence\": 0.7,\n  \"positionConsistency\": {\n    \"consistent\": true,\n    \"firstPassWinner\": \"B\",\n    \"secondPassWinner\": \"B\"\n  }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n  \"levels\": [\n    {\n      \"score\": 1,\n      \"label\": \"Poor\",\n      \"description\": \"Code is difficult to understand without significant effort\",\n      \"characteristics\": [\n        \"No meaningful variable or function names\",\n        \"No comments or documentation\",\n        \"Deeply nested or convoluted logic\"\n      ]\n    },\n    {\n      \"score\": 3,\n      \"label\": \"Adequate\",\n      \"description\": \"Code is understandable with some effort\",\n      \"characteristics\": [\n        \"Most variables have meaningful names\",\n        \"Basic comments present for complex sections\",\n        \"Logic is followable but could be cleaner\"\n      ]\n    },\n    {\n      \"score\": 5,\n      \"label\": \"Excellent\",\n      \"description\": \"Code is immediately clear and maintainable\",\n      \"characteristics\": [\n        \"All names are descriptive and consistent\",\n        \"Comprehensive documentation\",\n        \"Clean, modular structure\"\n      ]\n    }\n  ],\n  \"edgeCases\": [\n    {\n      \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n      \"guidance\": \"Score based on readability for domain experts, not general audience\"\n    }\n  ]\n}\n```\n\n## Guidelines\n\n1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25%\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Integration\n\nThis skill integrates with:\n\n- **context-fundamentals** - Evaluation prompts require effective context structure\n- **tool-design** - Evaluation tools need proper schemas and error handling\n- **context-optimization** - Evaluation prompts can be optimized for token efficiency\n- **evaluation** (foundational) - This skill extends the foundational evaluation concepts\n\n## References\n\nInternal reference:\n- LLM-as-Judge Implementation Patterns\n- Bias Mitigation Techniques\n- Metric Selection Guide\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https:\u002F\u002Feugeneyan.com\u002Fwriting\u002Fllm-evaluators\u002F)\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685)\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634)\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.17926)\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-24\n**Last Updated**: 2024-12-24\n**Author**: Muratcan Koylan\n**Version**: 1.0.0\n\n",{"name":5,"description":7},"main","b30820b3248f6676d5389f0ccabd4e5250c9c536","skills\u002Fadvanced-evaluation","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack","https:\u002F\u002Fgithub.com\u002Fkalash33","kalashpoddar.me","Tech-savvy Full Stack Developer 🚀 | Cybersecurity enthusiast 🔒 | Passionate about innovation and securing the digital world 🌐 | Let's code securely! 💻🔐","2026-05-20 04:11:22",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add kalash33\u002Fkstack --skill advanced-evaluation",{"tool":38,"command":39},"bunx","bunx skills add kalash33\u002Fkstack --skill advanced-evaluation",{"tool":41,"command":42},"pnpm","pnpm dlx skills add kalash33\u002Fkstack --skill advanced-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fadvanced-evaluation\u002FSKILL.md",true,[],[47,54,61,68,75,82],{"skillNo":48,"name":49,"nameZh":6,"description":50,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":51,"categories":52,"keywords":53,"publishedAt":19},"01M73CFERCR43JCAP606E41EFB","zustand-store-ts","Create Zustand stores with TypeScript, subscribeWithSelector middleware, and proper state\u002Faction separation. Use when building React state management, creating global stores, or implementing reacti...","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":55,"name":56,"nameZh":6,"description":57,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":19},"01M73CFCMFVTAQEMTZ5HR10X8V","zoom-automation","Automate Zoom meeting creation, management, recordings, webinars, and participant tracking via Rube MCP (Composio). Always search tools first for current schemas.","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fzoom-automation\u002FSKILL.md",[],[],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":19},"01M73CFAA7EBWXJVXDQ1X555MB","zoho-crm-automation","Automate Zoho CRM tasks via Rube MCP (Composio): create\u002Fupdate records, search contacts, manage leads, and convert leads. Always search tools first for current schemas.","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fzoho-crm-automation\u002FSKILL.md",[],[],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":19},"01M73CF80XCM9K81567MJ8QZFW","zod-validation-expert","Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fzod-validation-expert\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M73CF4YPMF3RA8ZJDXZSMD93","zeroize-audit","Detects missing zeroization of sensitive data in source code and identifies zeroization removed by compiler optimizations, with assembly-level analysis, and control-flow verification. Use for auditing C\u002FC++\u002FRust code handling secrets, keys, passwords, or other sensitive data.","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fzeroize-audit\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M73CF2S25EN3XNAMKY47XHPS","zendesk-automation","Automate Zendesk tasks via Rube MCP (Composio): tickets, users, organizations, replies. Always search tools first for current schemas.","https:\u002F\u002Fgithub.com\u002Fkalash33\u002Fkstack\u002Fblob\u002Fmain\u002Fskills\u002Fzendesk-automation\u002FSKILL.md",[],[]]