[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71N9DM5GF0CRPJKSEKBQJSG":3,"skill-same:01M71N9DM5GF0CRPJKSEKBQJSG":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":15,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":59,"repositoryHtmlUrl":60,"repositoryHomepageUrl":61,"ownerHtmlUrl":62,"ownerBlogUrl":63,"ownerDescription":6,"updatedAt":64,"install":65,"resources":78},"01M71N9DM5GF0CRPJKSEKBQJSG","llm-evaluation",null,"Use when building evaluation for an LLM app: automated metrics (BLEU, ROUGE, BERTScore, RAG retrieval), human rating, LLM-as-judge and regression baselines. Broad metric-level overview; for judge rubric design use advanced-evaluation, for agent pipelines use agent-evaluation.","monoes\u002Fmonomind","monoes","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F276744696?v=4","01M71KWF37AJ2NJBP3KWBVSDGT","01M71KWF4CSF8ADH0M6HA09VD2",23,"MIT","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fllm-evaluation\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fllm-evaluation\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,5,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"llm evaluation","Evaluation","as","metrics","metric","judge","rubric","Rubric Design","LLM-as-Judge","rag","human","when","design","testing","rating","regression","agent-evaluation","advanced","app","2026-11-10 10:56:23",false,"---\nname: llm-evaluation\ndescription: \"Use when building evaluation for an LLM app: automated metrics (BLEU, ROUGE, BERTScore, RAG retrieval), human rating, LLM-as-judge and regression baselines. Broad metric-level overview; for judge rubric design use advanced-evaluation, for agent pipelines use agent-evaluation.\"\ntags: [\"ai-ml\",\"testing\",\"llm\",\"evaluation\"]\ntools: []\nlicense: MIT\nsource: https:\u002F\u002Fgithub.com\u002Fwshobson\u002Fagents\nsource_path: \"plugins\u002Fllm-application-dev\u002Fskills\u002Fllm-evaluation\"\nsource_commit: 4236bb91f8395b0435f1d8b8baf9e8e4c69a8620\n---\n# LLM Evaluation\n\nMaster comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A\u002FB testing.\n\n## When to Use This Skill\n\n- Measuring LLM application performance systematically\n- Comparing different models or prompts\n- Detecting performance regressions before deployment\n- Validating improvements from prompt changes\n- Building confidence in production systems\n- Establishing baselines and tracking progress over time\n- Debugging unexpected model behavior\n\n## Core Evaluation Types\n\n### 1. Automated Metrics\n\nFast, repeatable, scalable evaluation using computed scores.\n\n**Text Generation:**\n\n- **BLEU**: N-gram overlap (translation)\n- **ROUGE**: Recall-oriented (summarization)\n- **METEOR**: Semantic similarity\n- **BERTScore**: Embedding-based similarity\n- **Perplexity**: Language model confidence\n\n**Classification:**\n\n- **Accuracy**: Percentage correct\n- **Precision\u002FRecall\u002FF1**: Class-specific performance\n- **Confusion Matrix**: Error patterns\n- **AUC-ROC**: Ranking quality\n\n**Retrieval (RAG):**\n\n- **MRR**: Mean Reciprocal Rank\n- **NDCG**: Normalized Discounted Cumulative Gain\n- **Precision@K**: Relevant in top K\n- **Recall@K**: Coverage in top K\n\n### 2. Human Evaluation\n\nManual assessment for quality aspects difficult to automate.\n\n**Dimensions:**\n\n- **Accuracy**: Factual correctness\n- **Coherence**: Logical flow\n- **Relevance**: Answers the question\n- **Fluency**: Natural language quality\n- **Safety**: No harmful content\n- **Helpfulness**: Useful to the user\n\n### 3. LLM-as-Judge\n\nUse stronger LLMs to evaluate weaker model outputs.\n\n**Approaches:**\n\n- **Pointwise**: Score individual responses\n- **Pairwise**: Compare two responses\n- **Reference-based**: Compare to gold standard\n- **Reference-free**: Judge without ground truth\n\n## Quick Start\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Callable\nimport numpy as np\n\n@dataclass\nclass Metric:\n    name: str\n    fn: Callable\n\n    @staticmethod\n    def accuracy():\n        return Metric(\"accuracy\", calculate_accuracy)\n\n    @staticmethod\n    def bleu():\n        return Metric(\"bleu\", calculate_bleu)\n\n    @staticmethod\n    def bertscore():\n        return Metric(\"bertscore\", calculate_bertscore)\n\n    @staticmethod\n    def custom(name: str, fn: Callable):\n        return Metric(name, fn)\n\nclass EvaluationSuite:\n    def __init__(self, metrics: list[Metric]):\n        self.metrics = metrics\n\n    async def evaluate(self, model, test_cases: list[dict]) -> dict:\n        results = {m.name: [] for m in self.metrics}\n\n        for test in test_cases:\n            prediction = await model.predict(test[\"input\"])\n\n            for metric in self.metrics:\n                score = metric.fn(\n                    prediction=prediction,\n                    reference=test.get(\"expected\"),\n                    context=test.get(\"context\")\n                )\n                results[metric.name].append(score)\n\n        return {\n            \"metrics\": {k: np.mean(v) for k, v in results.items()},\n            \"raw_scores\": results\n        }\n\n# Usage\nsuite = EvaluationSuite([\n    Metric.accuracy(),\n    Metric.bleu(),\n    Metric.bertscore(),\n    Metric.custom(\"groundedness\", check_groundedness)\n])\n\ntest_cases = [\n    {\n        \"input\": \"What is the capital of France?\",\n        \"expected\": \"Paris\",\n        \"context\": \"France is a country in Europe. Paris is its capital.\"\n    },\n]\n\nresults = await suite.evaluate(model=your_model, test_cases=test_cases)\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n","# LLM Evaluation\n\nMaster comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A\u002FB testing.\n\n## When to Use This Skill\n\n- Measuring LLM application performance systematically\n- Comparing different models or prompts\n- Detecting performance regressions before deployment\n- Validating improvements from prompt changes\n- Building confidence in production systems\n- Establishing baselines and tracking progress over time\n- Debugging unexpected model behavior\n\n## Core Evaluation Types\n\n### 1. Automated Metrics\n\nFast, repeatable, scalable evaluation using computed scores.\n\n**Text Generation:**\n\n- **BLEU**: N-gram overlap (translation)\n- **ROUGE**: Recall-oriented (summarization)\n- **METEOR**: Semantic similarity\n- **BERTScore**: Embedding-based similarity\n- **Perplexity**: Language model confidence\n\n**Classification:**\n\n- **Accuracy**: Percentage correct\n- **Precision\u002FRecall\u002FF1**: Class-specific performance\n- **Confusion Matrix**: Error patterns\n- **AUC-ROC**: Ranking quality\n\n**Retrieval (RAG):**\n\n- **MRR**: Mean Reciprocal Rank\n- **NDCG**: Normalized Discounted Cumulative Gain\n- **Precision@K**: Relevant in top K\n- **Recall@K**: Coverage in top K\n\n### 2. Human Evaluation\n\nManual assessment for quality aspects difficult to automate.\n\n**Dimensions:**\n\n- **Accuracy**: Factual correctness\n- **Coherence**: Logical flow\n- **Relevance**: Answers the question\n- **Fluency**: Natural language quality\n- **Safety**: No harmful content\n- **Helpfulness**: Useful to the user\n\n### 3. LLM-as-Judge\n\nUse stronger LLMs to evaluate weaker model outputs.\n\n**Approaches:**\n\n- **Pointwise**: Score individual responses\n- **Pairwise**: Compare two responses\n- **Reference-based**: Compare to gold standard\n- **Reference-free**: Judge without ground truth\n\n## Quick Start\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Callable\nimport numpy as np\n\n@dataclass\nclass Metric:\n    name: str\n    fn: Callable\n\n    @staticmethod\n    def accuracy():\n        return Metric(\"accuracy\", calculate_accuracy)\n\n    @staticmethod\n    def bleu():\n        return Metric(\"bleu\", calculate_bleu)\n\n    @staticmethod\n    def bertscore():\n        return Metric(\"bertscore\", calculate_bertscore)\n\n    @staticmethod\n    def custom(name: str, fn: Callable):\n        return Metric(name, fn)\n\nclass EvaluationSuite:\n    def __init__(self, metrics: list[Metric]):\n        self.metrics = metrics\n\n    async def evaluate(self, model, test_cases: list[dict]) -> dict:\n        results = {m.name: [] for m in self.metrics}\n\n        for test in test_cases:\n            prediction = await model.predict(test[\"input\"])\n\n            for metric in self.metrics:\n                score = metric.fn(\n                    prediction=prediction,\n                    reference=test.get(\"expected\"),\n                    context=test.get(\"context\")\n                )\n                results[metric.name].append(score)\n\n        return {\n            \"metrics\": {k: np.mean(v) for k, v in results.items()},\n            \"raw_scores\": results\n        }\n\n# Usage\nsuite = EvaluationSuite([\n    Metric.accuracy(),\n    Metric.bleu(),\n    Metric.bertscore(),\n    Metric.custom(\"groundedness\", check_groundedness)\n])\n\ntest_cases = [\n    {\n        \"input\": \"What is the capital of France?\",\n        \"expected\": \"Paris\",\n        \"context\": \"France is a country in Europe. Paris is its capital.\"\n    },\n]\n\nresults = await suite.evaluate(model=your_model, test_cases=test_cases)\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n",{"name":5,"tags":48,"tools":52,"source":53,"license":14,"description":7,"source_path":54,"source_commit":55},[49,37,50,51],"ai-ml","llm","evaluation",[],"https:\u002F\u002Fgithub.com\u002Fwshobson\u002Fagents","plugins\u002Fllm-application-dev\u002Fskills\u002Fllm-evaluation","4236bb91f8395b0435f1d8b8baf9e8e4c69a8620","main","f7cd4595d0f8959822163e14f9787117765c931a","packages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fllm-evaluation",2,"https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind","https:\u002F\u002Fmonoes.github.io\u002Fmonomind\u002F","https:\u002F\u002Fgithub.com\u002Fmonoes","https:\u002F\u002Fmonoes.me","2026-11-10 01:53:48",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add monoes\u002Fmonomind --skill llm-evaluation",{"tool":71,"command":72},"bunx","bunx skills add monoes\u002Fmonomind --skill llm-evaluation",{"tool":74,"command":75},"pnpm","pnpm dlx skills add monoes\u002Fmonomind --skill llm-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fllm-evaluation\u002FSKILL.md",true,[79,84],{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":15},"LICENSE.txt","LICENSE","许可证",1068,{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fdetails.md","REFERENCE","参考文档",14035,"https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fllm-evaluation\u002Freferences\u002Fdetails.md",[91,119,152,178,203,228],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":118},"01M71PB3SHWKMNJM6MD65VX8D6","zk-steward","Org role guidance for a ZK steward: engineer zero-knowledge proof systems with circuit design, SNARK vs STARK choice and security review. This is zero-knowledge cryptography, not Zettelkasten note-taking.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fzk-steward\u002FSKILL.md",[],[99,100,101,102,103,104,36,105,106,107,108,109,110,111,112,113,114,115,116,117],"zk","zk steward","steward","zero-knowledge","engineer","choice","knowledge","impact","query","snark","review","monomind","security review","stark","zettelkasten","cryptography","security","org","blockchain","2026-11-10 13:02:02",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":123,"sourceHtmlUrl":124,"categories":125,"keywords":134,"publishedAt":118},"01M71PB18SDG28WBKPKJNB814K","writing-plans","Use when you have a spec or requirements for a multi-step coding task and must write an implementation plan before touching code. Produces bite-sized TDD tasks with exact file paths, code and commands for an engineer with zero context.","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fwriting-plans\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fwriting-plans\u002FSKILL.md",[126,129],{"categoryNo":127,"categoryName":128,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":130,"categoryName":131,"taxonomyType":132,"taxonomyTypeDesc":133},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[135,136,137,103,138,139,35,140,107,141,142,143,144,145,146,147,148,149,150,151],"writing","writing plans","plans","superpowers","implementation plan","you","code","tasks","plan","product","write","commands","paths","suggest","coding","context","ZeRO",{"skillNo":153,"name":154,"nameZh":6,"description":155,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":95,"licenseFileUrl":156,"sourceHtmlUrl":157,"categories":158,"keywords":162,"publishedAt":118},"01M71PAXFM488X9ARXK0T04TF5","write-spec","Write a feature spec or PRD from a problem statement or feature idea. Use when turning a vague idea or user request into a structured document, scoping a feature with goals and non-goals, defining success metrics and acceptance criteria, or breaking a big ask into a phased spec.","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fwrite-spec\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fwrite-spec\u002FSKILL.md",[159],{"categoryNo":160,"categoryName":161,"taxonomyType":132,"taxonomyTypeDesc":133},"01JBSEEDCAT000000000000031","产品经理",[145,163,164,165,166,105,167,35,168,169,170,144,171,172,173,27,174,175,176,177],"write spec","spec","user request","problem","acceptance","ask","vague","work","request","scoping","idea","feature","problem statement","use","document",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":95,"licenseFileUrl":182,"sourceHtmlUrl":183,"categories":184,"keywords":188,"publishedAt":118},"01M71PASSZY800C0RPEARBS918","write-query","Write optimized SQL for your dialect with best practices. Use when translating a natural-language data need into SQL, building a multi-CTE query with joins and aggregations, optimizing a query against a large partitioned table, or getting dialect-specific syntax for Snowflake, BigQuery, Postgres, etc.","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fwrite-query\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fwrite-query\u002FSKILL.md",[185],{"categoryNo":186,"categoryName":187,"taxonomyType":132,"taxonomyTypeDesc":133},"01JBSEEDCAT000000000000033","数据分析师",[107,145,189,190,191,192,105,176,35,193,194,195,196,170,197,198,199,200,201,202],"write query","postgres","for","snowflake","best practices","natural-language","with","multi","github","sql","https","syntax","plugins","data",{"skillNo":204,"name":205,"nameZh":6,"description":206,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":207,"categories":208,"keywords":213,"publishedAt":118},"01M71PAPTX0T3PF6ETFVR96SJJ","workflow-automation","Org role guidance for workflow automation: build and maintain CI\u002FCD definitions in GitHub Actions or equivalent: triggers, jobs, matrices, caching. Covers path filters, dependency caching, parallel jobs, timeouts and fail-fast ordering.","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fworkflow-automation\u002FSKILL.md",[209,212],{"categoryNo":210,"categoryName":211,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[214,215,216,217,191,218,219,220,221,106,222,223,224,107,110,197,225,226,227,199],"Workflow Automation","automation","workflow","parallel","actions","ci","triggers","github actions","maintain","filters","caching","build","dependency","devops",{"skillNo":229,"name":230,"nameZh":6,"description":231,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":232,"categories":233,"keywords":235,"publishedAt":118},"01M71PAM19Z1XTV9NPCFSE2MY8","workflow-architect","Org role guidance for a workflow architect: map full workflow trees for systems, user journeys and agents, covering every branch and failure. Covers discovering workflows from code, handoff contracts, timeouts, recovery paths and observable state.","https:\u002F\u002Fgithub.com\u002Fmonoes\u002Fmonomind\u002Fblob\u002Fmain\u002Fpackages\u002F@monomind\u002Fcli\u002Forg-skills\u002Fworkflow-architect\u002FSKILL.md",[234],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[216,236,237,238,239,240,241,191,242,243,141,244,245,246,110,197,247,248,199,249],"workflow architect","architect","trees","user journeys","branch","planning","map","workflows","architecture","state","leadership","recovery","operations","systems"]