[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7F19EB2Z38DPFFJDY2YTYG3":3,"skill-same:01M7F19EB2Z38DPFFJDY2YTYG3":53},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":14,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7F19EB2Z38DPFFJDY2YTYG3","llm-evaluation",null,"Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.","shotcowboystyle\u002Fag-skills","shotcowboystyle","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F62471?v=4","01M7EZTCFCZJ6K3MYW8BPM4X6R","Curtis Blanton","01M7EZTCG1S5M04J90A931X9BX",0,"MIT","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fllm-evaluation\u002FSKILL.md",[],[],"2026-11-16 10:05:34",false,"---\nname: llm-evaluation\ndescription: Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.\nlast_verified: 2026-10-27\n---\n\n# LLM Evaluation\n\nMaster comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A\u002FB testing.\n\n## When to Use This Skill\n\n- Measuring LLM application performance systematically\n- Comparing different models or prompts\n- Detecting performance regressions before deployment\n- Validating improvements from prompt changes\n- Building confidence in production systems\n- Establishing baselines and tracking progress over time\n- Debugging unexpected model behavior\n\n## Core Evaluation Types\n\n### 1. Automated Metrics\n\nFast, repeatable, scalable evaluation using computed scores.\n\n**Text Generation:**\n\n- **BLEU**: N-gram overlap (translation)\n- **ROUGE**: Recall-oriented (summarization)\n- **METEOR**: Semantic similarity\n- **BERTScore**: Embedding-based similarity\n- **Perplexity**: Language model confidence\n\n**Classification:**\n\n- **Accuracy**: Percentage correct\n- **Precision\u002FRecall\u002FF1**: Class-specific performance\n- **Confusion Matrix**: Error patterns\n- **AUC-ROC**: Ranking quality\n\n**Retrieval (RAG):**\n\n- **MRR**: Mean Reciprocal Rank\n- **NDCG**: Normalized Discounted Cumulative Gain\n- **Precision@K**: Relevant in top K\n- **Recall@K**: Coverage in top K\n\n### 2. Human Evaluation\n\nManual assessment for quality aspects difficult to automate.\n\n**Dimensions:**\n\n- **Accuracy**: Factual correctness\n- **Coherence**: Logical flow\n- **Relevance**: Answers the question\n- **Fluency**: Natural language quality\n- **Safety**: No harmful content\n- **Helpfulness**: Useful to the user\n\n### 3. LLM-as-Judge\n\nUse stronger LLMs to evaluate weaker model outputs.\n\n**Approaches:**\n\n- **Pointwise**: Score individual responses\n- **Pairwise**: Compare two responses\n- **Reference-based**: Compare to gold standard\n- **Reference-free**: Judge without ground truth\n\n## Quick Start\n\nA self-contained harness: plain metric functions with one signature, an async model\ncallable, and mean scores per metric. Swap `fake_model` for your chain\n(`lambda q: chain.ainvoke(q)`) and add metrics from `references\u002Fdetails.md`.\n\n```python\nimport asyncio\nimport re\nfrom collections import Counter\nfrom statistics import mean\nfrom typing import Awaitable, Callable\n\n# Every metric: (prediction, reference, context) -> score in [0, 1]\nMetric = Callable[[str, str | None, str | None], float]\n\n\ndef _tokens(text: str) -> list[str]:\n    return re.findall(r\"\\w+\", text.lower())\n\n\ndef exact_match(prediction, reference, context=None) -> float:\n    return float(_tokens(prediction) == _tokens(reference or \"\"))\n\n\ndef token_f1(prediction, reference, context=None) -> float:\n    pred, ref = _tokens(prediction), _tokens(reference or \"\")\n    common = sum((Counter(pred) & Counter(ref)).values())\n    if not common:\n        return 0.0\n    precision, recall = common \u002F len(pred), common \u002F len(ref)\n    return 2 * precision * recall \u002F (precision + recall)\n\n\ndef context_overlap(prediction, reference=None, context=None) -> float:\n    \"\"\"Cheap groundedness proxy: share of answer tokens found in the context.\n    Use an NLI model or LLM judge (references\u002Fdetails.md) for real faithfulness.\"\"\"\n    pred = _tokens(prediction)\n    return sum(t in set(_tokens(context or \"\")) for t in pred) \u002F len(pred) if pred else 0.0\n\n\nasync def evaluate(\n    model: Callable[[str], Awaitable[str]],\n    test_cases: list[dict],\n    metrics: dict[str, Metric],\n) -> dict:\n    predictions = await asyncio.gather(*(model(t[\"input\"]) for t in test_cases))\n    raw = {\n        name: [fn(p, t.get(\"expected\"), t.get(\"context\")) for p, t in zip(predictions, test_cases)]\n        for name, fn in metrics.items()\n    }\n    return {\"metrics\": {name: mean(scores) for name, scores in raw.items()}, \"raw\": raw}\n\n\nasync def fake_model(question: str) -> str:  # replace with your application\n    return \"Paris\"\n\n\ntest_cases = [\n    {\n        \"input\": \"What is the capital of France?\",\n        \"expected\": \"Paris\",\n        \"context\": \"France is a country in Europe. Paris is its capital.\",\n    },\n]\n\nresults = asyncio.run(\n    evaluate(\n        fake_model,\n        test_cases,\n        {\"exact_match\": exact_match, \"token_f1\": token_f1, \"grounded\": context_overlap},\n    )\n)\nprint(results[\"metrics\"])  # {'exact_match': 1.0, 'token_f1': 1.0, 'grounded': 1.0}\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n\n","# LLM Evaluation\n\nMaster comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A\u002FB testing.\n\n## When to Use This Skill\n\n- Measuring LLM application performance systematically\n- Comparing different models or prompts\n- Detecting performance regressions before deployment\n- Validating improvements from prompt changes\n- Building confidence in production systems\n- Establishing baselines and tracking progress over time\n- Debugging unexpected model behavior\n\n## Core Evaluation Types\n\n### 1. Automated Metrics\n\nFast, repeatable, scalable evaluation using computed scores.\n\n**Text Generation:**\n\n- **BLEU**: N-gram overlap (translation)\n- **ROUGE**: Recall-oriented (summarization)\n- **METEOR**: Semantic similarity\n- **BERTScore**: Embedding-based similarity\n- **Perplexity**: Language model confidence\n\n**Classification:**\n\n- **Accuracy**: Percentage correct\n- **Precision\u002FRecall\u002FF1**: Class-specific performance\n- **Confusion Matrix**: Error patterns\n- **AUC-ROC**: Ranking quality\n\n**Retrieval (RAG):**\n\n- **MRR**: Mean Reciprocal Rank\n- **NDCG**: Normalized Discounted Cumulative Gain\n- **Precision@K**: Relevant in top K\n- **Recall@K**: Coverage in top K\n\n### 2. Human Evaluation\n\nManual assessment for quality aspects difficult to automate.\n\n**Dimensions:**\n\n- **Accuracy**: Factual correctness\n- **Coherence**: Logical flow\n- **Relevance**: Answers the question\n- **Fluency**: Natural language quality\n- **Safety**: No harmful content\n- **Helpfulness**: Useful to the user\n\n### 3. LLM-as-Judge\n\nUse stronger LLMs to evaluate weaker model outputs.\n\n**Approaches:**\n\n- **Pointwise**: Score individual responses\n- **Pairwise**: Compare two responses\n- **Reference-based**: Compare to gold standard\n- **Reference-free**: Judge without ground truth\n\n## Quick Start\n\nA self-contained harness: plain metric functions with one signature, an async model\ncallable, and mean scores per metric. Swap `fake_model` for your chain\n(`lambda q: chain.ainvoke(q)`) and add metrics from `references\u002Fdetails.md`.\n\n```python\nimport asyncio\nimport re\nfrom collections import Counter\nfrom statistics import mean\nfrom typing import Awaitable, Callable\n\n# Every metric: (prediction, reference, context) -> score in [0, 1]\nMetric = Callable[[str, str | None, str | None], float]\n\n\ndef _tokens(text: str) -> list[str]:\n    return re.findall(r\"\\w+\", text.lower())\n\n\ndef exact_match(prediction, reference, context=None) -> float:\n    return float(_tokens(prediction) == _tokens(reference or \"\"))\n\n\ndef token_f1(prediction, reference, context=None) -> float:\n    pred, ref = _tokens(prediction), _tokens(reference or \"\")\n    common = sum((Counter(pred) & Counter(ref)).values())\n    if not common:\n        return 0.0\n    precision, recall = common \u002F len(pred), common \u002F len(ref)\n    return 2 * precision * recall \u002F (precision + recall)\n\n\ndef context_overlap(prediction, reference=None, context=None) -> float:\n    \"\"\"Cheap groundedness proxy: share of answer tokens found in the context.\n    Use an NLI model or LLM judge (references\u002Fdetails.md) for real faithfulness.\"\"\"\n    pred = _tokens(prediction)\n    return sum(t in set(_tokens(context or \"\")) for t in pred) \u002F len(pred) if pred else 0.0\n\n\nasync def evaluate(\n    model: Callable[[str], Awaitable[str]],\n    test_cases: list[dict],\n    metrics: dict[str, Metric],\n) -> dict:\n    predictions = await asyncio.gather(*(model(t[\"input\"]) for t in test_cases))\n    raw = {\n        name: [fn(p, t.get(\"expected\"), t.get(\"context\")) for p, t in zip(predictions, test_cases)]\n        for name, fn in metrics.items()\n    }\n    return {\"metrics\": {name: mean(scores) for name, scores in raw.items()}, \"raw\": raw}\n\n\nasync def fake_model(question: str) -> str:  # replace with your application\n    return \"Paris\"\n\n\ntest_cases = [\n    {\n        \"input\": \"What is the capital of France?\",\n        \"expected\": \"Paris\",\n        \"context\": \"France is a country in Europe. Paris is its capital.\",\n    },\n]\n\nresults = asyncio.run(\n    evaluate(\n        fake_model,\n        test_cases,\n        {\"exact_match\": exact_match, \"token_f1\": token_f1, \"grounded\": context_overlap},\n    )\n)\nprint(results[\"metrics\"])  # {'exact_match': 1.0, 'token_f1': 1.0, 'grounded': 1.0}\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n\n",{"name":5,"description":7,"last_verified":24},1793059200000,"main","f846d9a97082ff30d613b55a085cef425d822473","plugins\u002Fllm-application-dev\u002Fskills\u002Fllm-evaluation","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle","https:\u002F\u002Fshotcowboystyle.github.io","Building smarter with AI, automation & modern tools. Exploring how far one person can go without a big team using today’s AI.","2026-11-10 04:58:27",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add shotcowboystyle\u002Fag-skills --skill llm-evaluation",{"tool":39,"command":40},"bunx","bunx skills add shotcowboystyle\u002Fag-skills --skill llm-evaluation",{"tool":42,"command":43},"pnpm","pnpm dlx skills add shotcowboystyle\u002Fag-skills --skill llm-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fllm-evaluation\u002FSKILL.md",true,[47],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"references\u002Fdetails.md","REFERENCE","参考文档",10796,"https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fllm-evaluation\u002Freferences\u002Fdetails.md",[54,60,67,74,81,88],{"skillNo":55,"name":56,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":19},"01M7F1YRY5S5QRJ1V0BJS130QX","no-description","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fno-description\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M7F1Y9BV2FJNQJ3NB32JS5CM","good-skill","Fixture skill that satisfies every rule.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fgood-skill\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M7F1XXBQ3NQRXZSS0KAGAR9V","broken-link","Links to a reference file that does not exist.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fbroken-link\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7F1XBX4R46NR9W0HXQ7XCDX","Bad_Name","Name violates the kebab-case rule and does not match the directory.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fbad-name\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7F1WNAF8WNEQ7S5HKT5VK0D","skill-authoring","Write, review, and validate portable agent skills in SKILL.md format. Use when creating a new skill, editing an existing one, or asked whether a skill follows the Agent Skills spec.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fskills\u002Fskill-authoring\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7F1W30TC0XFNSF7YBRGETK2","weigh-bags","Record scale readings for each bag — empty (tare), outbound, current, final — and reconcile them against the itemised inventory to find unaccounted weight. Use when the user gives a bag weight, says how heavy something is, weighs their luggage, or asks how close to the limit they are. Appends to bags.yaml and runs scripts\u002Freconcile.py.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Ftravel-packing-assistant\u002Fskills\u002Fweigh-bags\u002FSKILL.md",[],[]]