llm-evaluation
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
- License 许可
- MIT
- 最近更新
- 2026-11-10
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- llm-evaluation
- description
- Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
- last_verified
- 1793059200000
02 资源文件 1
- 参考文档references/details.md10.5 KB
03 SKILL.md 正文
LLM Evaluation
Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
When to Use This Skill
- Measuring LLM application performance systematically
- Comparing different models or prompts
- Detecting performance regressions before deployment
- Validating improvements from prompt changes
- Building confidence in production systems
- Establishing baselines and tracking progress over time
- Debugging unexpected model behavior
Core Evaluation Types
1. Automated Metrics
Fast, repeatable, scalable evaluation using computed scores.
Text Generation:
- BLEU: N-gram overlap (translation)
- ROUGE: Recall-oriented (summarization)
- METEOR: Semantic similarity
- BERTScore: Embedding-based similarity
- Perplexity: Language model confidence
Classification:
- Accuracy: Percentage correct
- Precision/Recall/F1: Class-specific performance
- Confusion Matrix: Error patterns
- AUC-ROC: Ranking quality
Retrieval (RAG):
- MRR: Mean Reciprocal Rank
- NDCG: Normalized Discounted Cumulative Gain
- Precision@K: Relevant in top K
- Recall@K: Coverage in top K
2. Human Evaluation
Manual assessment for quality aspects difficult to automate.
Dimensions:
- Accuracy: Factual correctness
- Coherence: Logical flow
- Relevance: Answers the question
- Fluency: Natural language quality
- Safety: No harmful content
- Helpfulness: Useful to the user
3. LLM-as-Judge
Use stronger LLMs to evaluate weaker model outputs.
Approaches:
- Pointwise: Score individual responses
- Pairwise: Compare two responses
- Reference-based: Compare to gold standard
- Reference-free: Judge without ground truth
Quick Start
A self-contained harness: plain metric functions with one signature, an async model
callable, and mean scores per metric. Swap fake_model for your chain
(lambda q: chain.ainvoke(q)) and add metrics from references/details.md.
import asyncio
import re
from collections import Counter
from statistics import mean
from typing import Awaitable, Callable
# Every metric: (prediction, reference, context) -> score in [0, 1]
Metric = Callable[[str, str | None, str | None], float]
def _tokens(text: str) -> list[str]:
return re.findall(r"\w+", text.lower())
def exact_match(prediction, reference, context=None) -> float:
return float(_tokens(prediction) == _tokens(reference or ""))
def token_f1(prediction, reference, context=None) -> float:
pred, ref = _tokens(prediction), _tokens(reference or "")
common = sum((Counter(pred) & Counter(ref)).values())
if not common:
return 0.0
precision, recall = common / len(pred), common / len(ref)
return 2 * precision * recall / (precision + recall)
def context_overlap(prediction, reference=None, context=None) -> float:
"""Cheap groundedness proxy: share of answer tokens found in the context.
Use an NLI model or LLM judge (references/details.md) for real faithfulness."""
pred = _tokens(prediction)
return sum(t in set(_tokens(context or "")) for t in pred) / len(pred) if pred else 0.0
async def evaluate(
model: Callable[[str], Awaitable[str]],
test_cases: list[dict],
metrics: dict[str, Metric],
) -> dict:
predictions = await asyncio.gather(*(model(t["input"]) for t in test_cases))
raw = {
name: [fn(p, t.get("expected"), t.get("context")) for p, t in zip(predictions, test_cases)]
for name, fn in metrics.items()
}
return {"metrics": {name: mean(scores) for name, scores in raw.items()}, "raw": raw}
async def fake_model(question: str) -> str: # replace with your application
return "Paris"
test_cases = [
{
"input": "What is the capital of France?",
"expected": "Paris",
"context": "France is a country in Europe. Paris is its capital.",
},
]
results = asyncio.run(
evaluate(
fake_model,
test_cases,
{"exact_match": exact_match, "token_f1": token_f1, "grounded": context_overlap},
)
)
print(results["metrics"]) # {'exact_match': 1.0, 'token_f1': 1.0, 'grounded': 1.0}
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
04 同仓库更多 Skills
no-description
暂无简介
good-skill
Fixture skill that satisfies every rule.
broken-link
Links to a reference file that does not exist.
Bad_Name
Name violates the kebab-case rule and does not match the directory.
skill-authoring
Write, review, and validate portable agent skills in SKILL.md format. Use when creating a new skill, editing an existing one, or asked whether a skill follows the Agent Skills spec.
weigh-bags
Record scale readings for each bag — empty (tare), outbound, current, final — and reconcile them against the itemised inventory to find unaccounted weight. Use when the user gives a bag weight, says how heavy something is, weighs their luggage, or asks how close to the limit they are. Appends to bags.yaml and runs scripts/reconcile.py.