[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78V3DMSMF2Y2ZGSFBB3B15G":3,"skill-same:01M78V3DMSMF2Y2ZGSFBB3B15G":81},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":14,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":48,"install":49,"resources":62},"01M78V3DMSMF2Y2ZGSFBB3B15G","evaluating-llms-harness",null,"Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.","mfrancime\u002Ftrident-archive","mfrancime","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F577553?v=4","01M75NH5W71WGY7NTS8YQ6VP8G","verzillion_kram","01M75NH5WTX2HBXJZWRP4DGAPK",0,"MIT","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: evaluating-llms-harness\ndescription: Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [lm-eval, transformers, vllm]\nmetadata:\n  hermes:\n    tags: [Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard]\n\n---\n\n# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\n**Issue: Evaluation too slow**\n\nUse vLLM backend:\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=model-name,tensor_parallel_size=2\n```\n\nOr reduce fewshot examples:\n```bash\n--num_fewshot 0  # Instead of 5\n```\n\nOr evaluate subset of MMLU:\n```bash\n--tasks mmlu_stem  # Only STEM subjects\n```\n\n**Issue: Out of memory**\n\nReduce batch size:\n```bash\n--batch_size 1  # Or --batch_size auto\n```\n\nUse quantization:\n```bash\n--model_args pretrained=model-name,load_in_8bit=True\n```\n\nEnable CPU offloading:\n```bash\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\n```\n\n**Issue: Different results than reported**\n\nCheck fewshot count:\n```bash\n--num_fewshot 5  # Most papers use 5-shot\n```\n\nCheck exact task name:\n```bash\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\n```\n\nVerify model and tokenizer match:\n```bash\n--model_args pretrained=model-name,tokenizer=same-model-name\n```\n\n**Issue: HumanEval not executing code**\n\nInstall execution dependencies:\n```bash\npip install human-eval\n```\n\nEnable code execution:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=model-name \\\n  --tasks humaneval \\\n  --allow_code_execution  # Required for HumanEval\n```\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\n- **VRAM**:\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\n  - 70B model: Requires multi-GPU or quantization\n- **Time** (7B model, single A100):\n  - HellaSwag: 10 minutes\n  - GSM8K: 5 minutes\n  - MMLU (full): 2 hours\n  - HumanEval: 20 minutes\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n\n\n","# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\n**Issue: Evaluation too slow**\n\nUse vLLM backend:\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=model-name,tensor_parallel_size=2\n```\n\nOr reduce fewshot examples:\n```bash\n--num_fewshot 0  # Instead of 5\n```\n\nOr evaluate subset of MMLU:\n```bash\n--tasks mmlu_stem  # Only STEM subjects\n```\n\n**Issue: Out of memory**\n\nReduce batch size:\n```bash\n--batch_size 1  # Or --batch_size auto\n```\n\nUse quantization:\n```bash\n--model_args pretrained=model-name,load_in_8bit=True\n```\n\nEnable CPU offloading:\n```bash\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\n```\n\n**Issue: Different results than reported**\n\nCheck fewshot count:\n```bash\n--num_fewshot 5  # Most papers use 5-shot\n```\n\nCheck exact task name:\n```bash\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\n```\n\nVerify model and tokenizer match:\n```bash\n--model_args pretrained=model-name,tokenizer=same-model-name\n```\n\n**Issue: HumanEval not executing code**\n\nInstall execution dependencies:\n```bash\npip install human-eval\n```\n\nEnable code execution:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=model-name \\\n  --tasks humaneval \\\n  --allow_code_execution  # Required for HumanEval\n```\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\n- **VRAM**:\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\n  - 70B model: Requires multi-GPU or quantization\n- **Time** (7B model, single A100):\n  - HellaSwag: 10 minutes\n  - GSM8K: 5 minutes\n  - MMLU (full): 2 hours\n  - HumanEval: 20 minutes\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"description":7,"dependencies":39},"Orchestra Research","1.0.0",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38],"Evaluation","LM Evaluation Harness","Benchmarking","MMLU","HumanEval","GSM8K","EleutherAI","Model Quality","Academic Benchmarks","Industry Standard",[40,41,42],"lm-eval","transformers","vllm","main","cef85894d3d9a4c9dabe5a90887d5ac59fe4ff0f","repos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive","https:\u002F\u002Fgithub.com\u002Fmfrancime","2026-09-11 19:19:01",{"source":8,"skillName":5,"commands":50,"prompt":60,"zipAvailable":61},[51,54,57],{"tool":52,"command":53},"npx","npx skills add mfrancime\u002Ftrident-archive --skill evaluating-llms-harness",{"tool":55,"command":56},"bunx","bunx skills add mfrancime\u002Ftrident-archive --skill evaluating-llms-harness",{"tool":58,"command":59},"pnpm","pnpm dlx skills add mfrancime\u002Ftrident-archive --skill evaluating-llms-harness","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness\u002FSKILL.md",true,[63,69,73,77],{"path":64,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fapi-evaluation.md","REFERENCE","参考文档",11114,"https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness\u002Freferences\u002Fapi-evaluation.md",{"path":70,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Fbenchmark-guide.md",10769,"https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness\u002Freferences\u002Fbenchmark-guide.md",{"path":74,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fcustom-tasks.md",13125,"https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness\u002Freferences\u002Fcustom-tasks.md",{"path":78,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fdistributed-eval.md",11426,"https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_hermes-agent_nousresearch_growing_ai\u002Fskills\u002Fmlops\u002Fevaluation\u002Flm-evaluation-harness\u002Freferences\u002Fdistributed-eval.md",[82,91,99,107,114,121],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":86,"licenseFileUrl":87,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M78VKWV0YXDY47JW2S4M19R1","skill-creator","Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_orcaclaw_sakana_personal_ai_assistant\u002Fskills\u002Fskill-creator\u002Flicense.txt","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_orcaclaw_sakana_personal_ai_assistant\u002Fskills\u002Fskill-creator\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M78VGSK0VQR8DFB181VS4MN8","prose","OpenProse VM skill pack. Activate on any `prose` command, .prose files, or OpenProse mentions; orchestrates multi-agent workflows.","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_orcaclaw_sakana_personal_ai_assistant\u002Fextensions\u002Fopen-prose\u002Fskills\u002Fprose\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_orcaclaw_sakana_personal_ai_assistant\u002Fextensions\u002Fopen-prose\u002Fskills\u002Fprose\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":86,"licenseFileUrl":103,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M78VFJ4N00J9RMF7X5AVZY7C","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_kilo-marketplace_skills_mcp_catalog\u002Fskills\u002Fwebapp-testing\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_kilo-marketplace_skills_mcp_catalog\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M78VF9C8TQ37WGS6WBT77BVW","vercel-react-best-practices","React and Next.js performance optimization guidelines from Vercel Engineering. This skill should be used when writing, reviewing, or refactoring React\u002FNext.js code to ensure optimal performance patterns. Triggers on tasks involving React components, Next.js pages, data fetching, bundle optimization, or performance improvements.","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_kilo-marketplace_skills_mcp_catalog\u002Fskills\u002Fvercel-react-best-practices\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M78VF5VFD2CA77AM9AVBXR7C","vercel-deploy","Deploy applications and websites to Vercel. Use this skill when the user requests deployment actions such as \"Deploy my app\", \"Deploy this to production\", \"Create a preview deployment\", \"Deploy and give me the link\", or \"Push this live\". No authentication required - returns preview URL and claimable deployment link.","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_kilo-marketplace_skills_mcp_catalog\u002Fskills\u002Fvercel-deploy\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":19},"01M78VF0JF37J4KCM4Y30Z9E55","vercel-composition-patterns","React composition patterns that scale. Use when refactoring components with boolean prop proliferation, building flexible component libraries, or designing reusable APIs. Triggers on tasks involving compound components, render props, context providers, or component architecture. Includes React 19 API changes.","https:\u002F\u002Fgithub.com\u002Fmfrancime\u002Ftrident-archive\u002Fblob\u002Fmain\u002Frepos\u002Ftrident_kilo-marketplace_skills_mcp_catalog\u002Fskills\u002Fvercel-composition-patterns\u002FSKILL.md",[],[]]