[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6DEVSPQMKM9SJGDDP9QD0XS":3,"skill-same:01M6DEVSPQMKM9SJGDDP9QD0XS":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":61,"commitSha":62,"skillDirectory":63,"forksCount":64,"repositoryHtmlUrl":65,"repositoryHomepageUrl":6,"ownerHtmlUrl":66,"ownerBlogUrl":6,"ownerDescription":67,"updatedAt":68,"install":69,"resources":82},"01M6DEVSPQMKM9SJGDDP9QD0XS","evaluating-llms-harness",null,"Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.","zapabob\u002FSkills","zapabob","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F130830376?v=4","01M6DBMYZAGRTFQDSQYW3W764P","峯岸 亮","01M6DBMZ3GKDSJ01NZ9XTZD7JD",2,"MIT","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002F41\u002F41e2dd15ab0d1744355ed1e5763d3f396ba5efbcd3a179c745e85c3e364aefb0\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"harness","evaluating llms harness","llms","Transformers","across","benchmarks","models","use","results","eval","quality","model","vLLM","HuggingFace","MMLU","Evaluation","reporting","training","benchmarking","research","2026-11-10 10:12:30",false,"---\r\nname: evaluating-llms-harness\r\ndescription: Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.\r\nversion: 1.0.0\r\nauthor: Orchestra Research\r\nlicense: MIT\r\ndependencies: [lm-eval, transformers, vllm]\r\nmetadata:\r\n  hermes:\r\n    tags: [Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard]\r\n\r\n---\r\n\r\n# lm-evaluation-harness - LLM Benchmarking\r\n\r\n## Quick start\r\n\r\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\r\n\r\n**Installation**:\r\n```bash\r\npip install lm-eval\r\n```\r\n\r\n**Evaluate any HuggingFace model**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu,gsm8k,hellaswag \\\r\n  --device cuda:0 \\\r\n  --batch_size 8\r\n```\r\n\r\n**View available tasks**:\r\n```bash\r\nlm_eval --tasks list\r\n```\r\n\r\n## Common workflows\r\n\r\n### Workflow 1: Standard benchmark evaluation\r\n\r\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\r\n\r\nCopy this checklist:\r\n\r\n```\r\nBenchmark Evaluation:\r\n- [ ] Step 1: Choose benchmark suite\r\n- [ ] Step 2: Configure model\r\n- [ ] Step 3: Run evaluation\r\n- [ ] Step 4: Analyze results\r\n```\r\n\r\n**Step 1: Choose benchmark suite**\r\n\r\n**Core reasoning benchmarks**:\r\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\r\n- **GSM8K** - Grade school math word problems\r\n- **HellaSwag** - Common sense reasoning\r\n- **TruthfulQA** - Truthfulness and factuality\r\n- **ARC** (AI2 Reasoning Challenge) - Science questions\r\n\r\n**Code benchmarks**:\r\n- **HumanEval** - Python code generation (164 problems)\r\n- **MBPP** (Mostly Basic Python Problems) - Python coding\r\n\r\n**Standard suite** (recommended for model releases):\r\n```bash\r\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\r\n```\r\n\r\n**Step 2: Configure model**\r\n\r\n**HuggingFace model**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\r\n  --tasks mmlu \\\r\n  --device cuda:0 \\\r\n  --batch_size auto  # Auto-detect optimal batch size\r\n```\r\n\r\n**Quantized model (4-bit\u002F8-bit)**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\r\n  --tasks mmlu \\\r\n  --device cuda:0\r\n```\r\n\r\n**Custom checkpoint**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\r\n  --tasks mmlu \\\r\n  --device cuda:0\r\n```\r\n\r\n**Step 3: Run evaluation**\r\n\r\n```bash\r\n# Full MMLU evaluation (57 subjects)\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu \\\r\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\r\n  --batch_size 8 \\\r\n  --output_path results\u002F \\\r\n  --log_samples  # Save individual predictions\r\n\r\n# Multiple benchmarks at once\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\r\n  --num_fewshot 5 \\\r\n  --batch_size 8 \\\r\n  --output_path results\u002Fllama2-7b-eval.json\r\n```\r\n\r\n**Step 4: Analyze results**\r\n\r\nResults saved to `results\u002Fllama2-7b-eval.json`:\r\n\r\n```json\r\n{\r\n  \"results\": {\r\n    \"mmlu\": {\r\n      \"acc\": 0.459,\r\n      \"acc_stderr\": 0.004\r\n    },\r\n    \"gsm8k\": {\r\n      \"exact_match\": 0.142,\r\n      \"exact_match_stderr\": 0.006\r\n    },\r\n    \"hellaswag\": {\r\n      \"acc_norm\": 0.765,\r\n      \"acc_norm_stderr\": 0.004\r\n    }\r\n  },\r\n  \"config\": {\r\n    \"model\": \"hf\",\r\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\r\n    \"num_fewshot\": 5\r\n  }\r\n}\r\n```\r\n\r\n### Workflow 2: Track training progress\r\n\r\nEvaluate checkpoints during training.\r\n\r\n```\r\nTraining Progress Tracking:\r\n- [ ] Step 1: Set up periodic evaluation\r\n- [ ] Step 2: Choose quick benchmarks\r\n- [ ] Step 3: Automate evaluation\r\n- [ ] Step 4: Plot learning curves\r\n```\r\n\r\n**Step 1: Set up periodic evaluation**\r\n\r\nEvaluate every N training steps:\r\n\r\n```bash\r\n#!\u002Fbin\u002Fbash\r\n# eval_checkpoint.sh\r\n\r\nCHECKPOINT_DIR=$1\r\nSTEP=$2\r\n\r\nlm_eval --model hf \\\r\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\r\n  --tasks gsm8k,hellaswag \\\r\n  --num_fewshot 0 \\  # 0-shot for speed\r\n  --batch_size 16 \\\r\n  --output_path results\u002Fstep-$STEP.json\r\n```\r\n\r\n**Step 2: Choose quick benchmarks**\r\n\r\nFast benchmarks for frequent evaluation:\r\n- **HellaSwag**: ~10 minutes on 1 GPU\r\n- **GSM8K**: ~5 minutes\r\n- **PIQA**: ~2 minutes\r\n\r\nAvoid for frequent eval (too slow):\r\n- **MMLU**: ~2 hours (57 subjects)\r\n- **HumanEval**: Requires code execution\r\n\r\n**Step 3: Automate evaluation**\r\n\r\nIntegrate with training script:\r\n\r\n```python\r\n# In training loop\r\nif step % eval_interval == 0:\r\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\r\n\r\n    # Run evaluation\r\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\r\n```\r\n\r\nOr use PyTorch Lightning callbacks:\r\n\r\n```python\r\nfrom pytorch_lightning import Callback\r\n\r\nclass EvalHarnessCallback(Callback):\r\n    def on_validation_epoch_end(self, trainer, pl_module):\r\n        step = trainer.global_step\r\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\r\n\r\n        # Save checkpoint\r\n        trainer.save_checkpoint(checkpoint_path)\r\n\r\n        # Run lm-eval\r\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\r\n```\r\n\r\n**Step 4: Plot learning curves**\r\n\r\n```python\r\nimport json\r\nimport matplotlib.pyplot as plt\r\n\r\n# Load all results\r\nsteps = []\r\nmmlu_scores = []\r\n\r\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\r\n    with open(file) as f:\r\n        data = json.load(f)\r\n        step = int(file.split(\"-\")[1].split(\".\")[0])\r\n        steps.append(step)\r\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\r\n\r\n# Plot\r\nplt.plot(steps, mmlu_scores)\r\nplt.xlabel(\"Training Step\")\r\nplt.ylabel(\"MMLU Accuracy\")\r\nplt.title(\"Training Progress\")\r\nplt.savefig(\"training_curve.png\")\r\n```\r\n\r\n### Workflow 3: Compare multiple models\r\n\r\nBenchmark suite for model comparison.\r\n\r\n```\r\nModel Comparison:\r\n- [ ] Step 1: Define model list\r\n- [ ] Step 2: Run evaluations\r\n- [ ] Step 3: Generate comparison table\r\n```\r\n\r\n**Step 1: Define model list**\r\n\r\n```bash\r\n# models.txt\r\nmeta-llama\u002FLlama-2-7b-hf\r\nmeta-llama\u002FLlama-2-13b-hf\r\nmistralai\u002FMistral-7B-v0.1\r\nmicrosoft\u002Fphi-2\r\n```\r\n\r\n**Step 2: Run evaluations**\r\n\r\n```bash\r\n#!\u002Fbin\u002Fbash\r\n# eval_all_models.sh\r\n\r\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\r\n\r\nwhile read model; do\r\n    echo \"Evaluating $model\"\r\n\r\n    # Extract model name for output file\r\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\r\n\r\n    lm_eval --model hf \\\r\n      --model_args pretrained=$model,dtype=bfloat16 \\\r\n      --tasks $TASKS \\\r\n      --num_fewshot 5 \\\r\n      --batch_size auto \\\r\n      --output_path results\u002F$model_name.json\r\n\r\ndone \u003C models.txt\r\n```\r\n\r\n**Step 3: Generate comparison table**\r\n\r\n```python\r\nimport json\r\nimport pandas as pd\r\n\r\nmodels = [\r\n    \"meta-llama-Llama-2-7b-hf\",\r\n    \"meta-llama-Llama-2-13b-hf\",\r\n    \"mistralai-Mistral-7B-v0.1\",\r\n    \"microsoft-phi-2\"\r\n]\r\n\r\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\r\n\r\nresults = []\r\nfor model in models:\r\n    with open(f\"results\u002F{model}.json\") as f:\r\n        data = json.load(f)\r\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\r\n        for task in tasks:\r\n            # Get primary metric for each task\r\n            metrics = data[\"results\"][task]\r\n            if \"acc\" in metrics:\r\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\r\n            elif \"exact_match\" in metrics:\r\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\r\n        results.append(row)\r\n\r\ndf = pd.DataFrame(results)\r\nprint(df.to_markdown(index=False))\r\n```\r\n\r\nOutput:\r\n```\r\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\r\n|------------------------|-------|-------|-----------|------------|\r\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\r\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\r\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\r\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\r\n```\r\n\r\n### Workflow 4: Evaluate with vLLM (faster inference)\r\n\r\nUse vLLM backend for 5-10x faster evaluation.\r\n\r\n```\r\nvLLM Evaluation:\r\n- [ ] Step 1: Install vLLM\r\n- [ ] Step 2: Configure vLLM backend\r\n- [ ] Step 3: Run evaluation\r\n```\r\n\r\n**Step 1: Install vLLM**\r\n\r\n```bash\r\npip install vllm\r\n```\r\n\r\n**Step 2: Configure vLLM backend**\r\n\r\n```bash\r\nlm_eval --model vllm \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\r\n  --tasks mmlu \\\r\n  --batch_size auto\r\n```\r\n\r\n**Step 3: Run evaluation**\r\n\r\nvLLM is 5-10× faster than standard HuggingFace:\r\n\r\n```bash\r\n# Standard HF: ~2 hours for MMLU on 7B model\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu \\\r\n  --batch_size 8\r\n\r\n# vLLM: ~15-20 minutes for MMLU on 7B model\r\nlm_eval --model vllm \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\r\n  --tasks mmlu \\\r\n  --batch_size auto\r\n```\r\n\r\n## When to use vs alternatives\r\n\r\n**Use lm-evaluation-harness when:**\r\n- Benchmarking models for academic papers\r\n- Comparing model quality across standard tasks\r\n- Tracking training progress\r\n- Reporting standardized metrics (everyone uses same prompts)\r\n- Need reproducible evaluation\r\n\r\n**Use alternatives instead:**\r\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\r\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\r\n- **MT-Bench**: Conversational multi-turn evaluation\r\n- **Custom scripts**: Domain-specific evaluation\r\n\r\n## Common issues\r\n\r\n**Issue: Evaluation too slow**\r\n\r\nUse vLLM backend:\r\n```bash\r\nlm_eval --model vllm \\\r\n  --model_args pretrained=model-name,tensor_parallel_size=2\r\n```\r\n\r\nOr reduce fewshot examples:\r\n```bash\r\n--num_fewshot 0  # Instead of 5\r\n```\r\n\r\nOr evaluate subset of MMLU:\r\n```bash\r\n--tasks mmlu_stem  # Only STEM subjects\r\n```\r\n\r\n**Issue: Out of memory**\r\n\r\nReduce batch size:\r\n```bash\r\n--batch_size 1  # Or --batch_size auto\r\n```\r\n\r\nUse quantization:\r\n```bash\r\n--model_args pretrained=model-name,load_in_8bit=True\r\n```\r\n\r\nEnable CPU offloading:\r\n```bash\r\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\r\n```\r\n\r\n**Issue: Different results than reported**\r\n\r\nCheck fewshot count:\r\n```bash\r\n--num_fewshot 5  # Most papers use 5-shot\r\n```\r\n\r\nCheck exact task name:\r\n```bash\r\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\r\n```\r\n\r\nVerify model and tokenizer match:\r\n```bash\r\n--model_args pretrained=model-name,tokenizer=same-model-name\r\n```\r\n\r\n**Issue: HumanEval not executing code**\r\n\r\nInstall execution dependencies:\r\n```bash\r\npip install human-eval\r\n```\r\n\r\nEnable code execution:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=model-name \\\r\n  --tasks humaneval \\\r\n  --allow_code_execution  # Required for HumanEval\r\n```\r\n\r\n## Advanced topics\r\n\r\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\r\n\r\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\r\n\r\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\r\n\r\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\r\n\r\n## Hardware requirements\r\n\r\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\r\n- **VRAM**:\r\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\r\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\r\n  - 70B model: Requires multi-GPU or quantization\r\n- **Time** (7B model, single A100):\r\n  - HellaSwag: 10 minutes\r\n  - GSM8K: 5 minutes\r\n  - MMLU (full): 2 hours\r\n  - HumanEval: 20 minutes\r\n\r\n## Resources\r\n\r\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\r\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\r\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\r\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\r\n","# lm-evaluation-harness - LLM Benchmarking\r\n\r\n## Quick start\r\n\r\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\r\n\r\n**Installation**:\r\n```bash\r\npip install lm-eval\r\n```\r\n\r\n**Evaluate any HuggingFace model**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu,gsm8k,hellaswag \\\r\n  --device cuda:0 \\\r\n  --batch_size 8\r\n```\r\n\r\n**View available tasks**:\r\n```bash\r\nlm_eval --tasks list\r\n```\r\n\r\n## Common workflows\r\n\r\n### Workflow 1: Standard benchmark evaluation\r\n\r\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\r\n\r\nCopy this checklist:\r\n\r\n```\r\nBenchmark Evaluation:\r\n- [ ] Step 1: Choose benchmark suite\r\n- [ ] Step 2: Configure model\r\n- [ ] Step 3: Run evaluation\r\n- [ ] Step 4: Analyze results\r\n```\r\n\r\n**Step 1: Choose benchmark suite**\r\n\r\n**Core reasoning benchmarks**:\r\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\r\n- **GSM8K** - Grade school math word problems\r\n- **HellaSwag** - Common sense reasoning\r\n- **TruthfulQA** - Truthfulness and factuality\r\n- **ARC** (AI2 Reasoning Challenge) - Science questions\r\n\r\n**Code benchmarks**:\r\n- **HumanEval** - Python code generation (164 problems)\r\n- **MBPP** (Mostly Basic Python Problems) - Python coding\r\n\r\n**Standard suite** (recommended for model releases):\r\n```bash\r\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\r\n```\r\n\r\n**Step 2: Configure model**\r\n\r\n**HuggingFace model**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\r\n  --tasks mmlu \\\r\n  --device cuda:0 \\\r\n  --batch_size auto  # Auto-detect optimal batch size\r\n```\r\n\r\n**Quantized model (4-bit\u002F8-bit)**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\r\n  --tasks mmlu \\\r\n  --device cuda:0\r\n```\r\n\r\n**Custom checkpoint**:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\r\n  --tasks mmlu \\\r\n  --device cuda:0\r\n```\r\n\r\n**Step 3: Run evaluation**\r\n\r\n```bash\r\n# Full MMLU evaluation (57 subjects)\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu \\\r\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\r\n  --batch_size 8 \\\r\n  --output_path results\u002F \\\r\n  --log_samples  # Save individual predictions\r\n\r\n# Multiple benchmarks at once\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\r\n  --num_fewshot 5 \\\r\n  --batch_size 8 \\\r\n  --output_path results\u002Fllama2-7b-eval.json\r\n```\r\n\r\n**Step 4: Analyze results**\r\n\r\nResults saved to `results\u002Fllama2-7b-eval.json`:\r\n\r\n```json\r\n{\r\n  \"results\": {\r\n    \"mmlu\": {\r\n      \"acc\": 0.459,\r\n      \"acc_stderr\": 0.004\r\n    },\r\n    \"gsm8k\": {\r\n      \"exact_match\": 0.142,\r\n      \"exact_match_stderr\": 0.006\r\n    },\r\n    \"hellaswag\": {\r\n      \"acc_norm\": 0.765,\r\n      \"acc_norm_stderr\": 0.004\r\n    }\r\n  },\r\n  \"config\": {\r\n    \"model\": \"hf\",\r\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\r\n    \"num_fewshot\": 5\r\n  }\r\n}\r\n```\r\n\r\n### Workflow 2: Track training progress\r\n\r\nEvaluate checkpoints during training.\r\n\r\n```\r\nTraining Progress Tracking:\r\n- [ ] Step 1: Set up periodic evaluation\r\n- [ ] Step 2: Choose quick benchmarks\r\n- [ ] Step 3: Automate evaluation\r\n- [ ] Step 4: Plot learning curves\r\n```\r\n\r\n**Step 1: Set up periodic evaluation**\r\n\r\nEvaluate every N training steps:\r\n\r\n```bash\r\n#!\u002Fbin\u002Fbash\r\n# eval_checkpoint.sh\r\n\r\nCHECKPOINT_DIR=$1\r\nSTEP=$2\r\n\r\nlm_eval --model hf \\\r\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\r\n  --tasks gsm8k,hellaswag \\\r\n  --num_fewshot 0 \\  # 0-shot for speed\r\n  --batch_size 16 \\\r\n  --output_path results\u002Fstep-$STEP.json\r\n```\r\n\r\n**Step 2: Choose quick benchmarks**\r\n\r\nFast benchmarks for frequent evaluation:\r\n- **HellaSwag**: ~10 minutes on 1 GPU\r\n- **GSM8K**: ~5 minutes\r\n- **PIQA**: ~2 minutes\r\n\r\nAvoid for frequent eval (too slow):\r\n- **MMLU**: ~2 hours (57 subjects)\r\n- **HumanEval**: Requires code execution\r\n\r\n**Step 3: Automate evaluation**\r\n\r\nIntegrate with training script:\r\n\r\n```python\r\n# In training loop\r\nif step % eval_interval == 0:\r\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\r\n\r\n    # Run evaluation\r\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\r\n```\r\n\r\nOr use PyTorch Lightning callbacks:\r\n\r\n```python\r\nfrom pytorch_lightning import Callback\r\n\r\nclass EvalHarnessCallback(Callback):\r\n    def on_validation_epoch_end(self, trainer, pl_module):\r\n        step = trainer.global_step\r\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\r\n\r\n        # Save checkpoint\r\n        trainer.save_checkpoint(checkpoint_path)\r\n\r\n        # Run lm-eval\r\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\r\n```\r\n\r\n**Step 4: Plot learning curves**\r\n\r\n```python\r\nimport json\r\nimport matplotlib.pyplot as plt\r\n\r\n# Load all results\r\nsteps = []\r\nmmlu_scores = []\r\n\r\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\r\n    with open(file) as f:\r\n        data = json.load(f)\r\n        step = int(file.split(\"-\")[1].split(\".\")[0])\r\n        steps.append(step)\r\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\r\n\r\n# Plot\r\nplt.plot(steps, mmlu_scores)\r\nplt.xlabel(\"Training Step\")\r\nplt.ylabel(\"MMLU Accuracy\")\r\nplt.title(\"Training Progress\")\r\nplt.savefig(\"training_curve.png\")\r\n```\r\n\r\n### Workflow 3: Compare multiple models\r\n\r\nBenchmark suite for model comparison.\r\n\r\n```\r\nModel Comparison:\r\n- [ ] Step 1: Define model list\r\n- [ ] Step 2: Run evaluations\r\n- [ ] Step 3: Generate comparison table\r\n```\r\n\r\n**Step 1: Define model list**\r\n\r\n```bash\r\n# models.txt\r\nmeta-llama\u002FLlama-2-7b-hf\r\nmeta-llama\u002FLlama-2-13b-hf\r\nmistralai\u002FMistral-7B-v0.1\r\nmicrosoft\u002Fphi-2\r\n```\r\n\r\n**Step 2: Run evaluations**\r\n\r\n```bash\r\n#!\u002Fbin\u002Fbash\r\n# eval_all_models.sh\r\n\r\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\r\n\r\nwhile read model; do\r\n    echo \"Evaluating $model\"\r\n\r\n    # Extract model name for output file\r\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\r\n\r\n    lm_eval --model hf \\\r\n      --model_args pretrained=$model,dtype=bfloat16 \\\r\n      --tasks $TASKS \\\r\n      --num_fewshot 5 \\\r\n      --batch_size auto \\\r\n      --output_path results\u002F$model_name.json\r\n\r\ndone \u003C models.txt\r\n```\r\n\r\n**Step 3: Generate comparison table**\r\n\r\n```python\r\nimport json\r\nimport pandas as pd\r\n\r\nmodels = [\r\n    \"meta-llama-Llama-2-7b-hf\",\r\n    \"meta-llama-Llama-2-13b-hf\",\r\n    \"mistralai-Mistral-7B-v0.1\",\r\n    \"microsoft-phi-2\"\r\n]\r\n\r\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\r\n\r\nresults = []\r\nfor model in models:\r\n    with open(f\"results\u002F{model}.json\") as f:\r\n        data = json.load(f)\r\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\r\n        for task in tasks:\r\n            # Get primary metric for each task\r\n            metrics = data[\"results\"][task]\r\n            if \"acc\" in metrics:\r\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\r\n            elif \"exact_match\" in metrics:\r\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\r\n        results.append(row)\r\n\r\ndf = pd.DataFrame(results)\r\nprint(df.to_markdown(index=False))\r\n```\r\n\r\nOutput:\r\n```\r\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\r\n|------------------------|-------|-------|-----------|------------|\r\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\r\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\r\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\r\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\r\n```\r\n\r\n### Workflow 4: Evaluate with vLLM (faster inference)\r\n\r\nUse vLLM backend for 5-10x faster evaluation.\r\n\r\n```\r\nvLLM Evaluation:\r\n- [ ] Step 1: Install vLLM\r\n- [ ] Step 2: Configure vLLM backend\r\n- [ ] Step 3: Run evaluation\r\n```\r\n\r\n**Step 1: Install vLLM**\r\n\r\n```bash\r\npip install vllm\r\n```\r\n\r\n**Step 2: Configure vLLM backend**\r\n\r\n```bash\r\nlm_eval --model vllm \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\r\n  --tasks mmlu \\\r\n  --batch_size auto\r\n```\r\n\r\n**Step 3: Run evaluation**\r\n\r\nvLLM is 5-10× faster than standard HuggingFace:\r\n\r\n```bash\r\n# Standard HF: ~2 hours for MMLU on 7B model\r\nlm_eval --model hf \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\r\n  --tasks mmlu \\\r\n  --batch_size 8\r\n\r\n# vLLM: ~15-20 minutes for MMLU on 7B model\r\nlm_eval --model vllm \\\r\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\r\n  --tasks mmlu \\\r\n  --batch_size auto\r\n```\r\n\r\n## When to use vs alternatives\r\n\r\n**Use lm-evaluation-harness when:**\r\n- Benchmarking models for academic papers\r\n- Comparing model quality across standard tasks\r\n- Tracking training progress\r\n- Reporting standardized metrics (everyone uses same prompts)\r\n- Need reproducible evaluation\r\n\r\n**Use alternatives instead:**\r\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\r\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\r\n- **MT-Bench**: Conversational multi-turn evaluation\r\n- **Custom scripts**: Domain-specific evaluation\r\n\r\n## Common issues\r\n\r\n**Issue: Evaluation too slow**\r\n\r\nUse vLLM backend:\r\n```bash\r\nlm_eval --model vllm \\\r\n  --model_args pretrained=model-name,tensor_parallel_size=2\r\n```\r\n\r\nOr reduce fewshot examples:\r\n```bash\r\n--num_fewshot 0  # Instead of 5\r\n```\r\n\r\nOr evaluate subset of MMLU:\r\n```bash\r\n--tasks mmlu_stem  # Only STEM subjects\r\n```\r\n\r\n**Issue: Out of memory**\r\n\r\nReduce batch size:\r\n```bash\r\n--batch_size 1  # Or --batch_size auto\r\n```\r\n\r\nUse quantization:\r\n```bash\r\n--model_args pretrained=model-name,load_in_8bit=True\r\n```\r\n\r\nEnable CPU offloading:\r\n```bash\r\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\r\n```\r\n\r\n**Issue: Different results than reported**\r\n\r\nCheck fewshot count:\r\n```bash\r\n--num_fewshot 5  # Most papers use 5-shot\r\n```\r\n\r\nCheck exact task name:\r\n```bash\r\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\r\n```\r\n\r\nVerify model and tokenizer match:\r\n```bash\r\n--model_args pretrained=model-name,tokenizer=same-model-name\r\n```\r\n\r\n**Issue: HumanEval not executing code**\r\n\r\nInstall execution dependencies:\r\n```bash\r\npip install human-eval\r\n```\r\n\r\nEnable code execution:\r\n```bash\r\nlm_eval --model hf \\\r\n  --model_args pretrained=model-name \\\r\n  --tasks humaneval \\\r\n  --allow_code_execution  # Required for HumanEval\r\n```\r\n\r\n## Advanced topics\r\n\r\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\r\n\r\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\r\n\r\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\r\n\r\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\r\n\r\n## Hardware requirements\r\n\r\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\r\n- **VRAM**:\r\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\r\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\r\n  - 70B model: Requires multi-GPU or quantization\r\n- **Time** (7B model, single A100):\r\n  - HellaSwag: 10 minutes\r\n  - GSM8K: 5 minutes\r\n  - MMLU (full): 2 hours\r\n  - HumanEval: 20 minutes\r\n\r\n## Resources\r\n\r\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\r\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\r\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\r\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\r\n",{"name":5,"author":44,"license":15,"version":45,"metadata":46,"description":7,"dependencies":57},"Orchestra Research","1.0.0",{"hermes":47},{"tags":48},[34,49,50,33,51,52,53,54,55,56],"LM Evaluation Harness","Benchmarking","HumanEval","GSM8K","EleutherAI","Model Quality","Academic Benchmarks","Industry Standard",[58,59,60],"lm-eval","transformers","vllm","main","0d1434f89f4bc6b07e9d272accd8c1013768820f","archive\u002Fskill-md-reviewed\u002Fsha256\u002F41\u002F41e2dd15ab0d1744355ed1e5763d3f396ba5efbcd3a179c745e85c3e364aefb0",0,"https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills","https:\u002F\u002Fgithub.com\u002Fzapabob","The Open University of JAPAN　2019-now\r\n\r\nTokyo University of Science　2007-2010","2026-10-26 16:29:01",{"source":8,"skillName":5,"commands":70,"prompt":80,"zipAvailable":81},[71,74,77],{"tool":72,"command":73},"npx","npx skills add zapabob\u002FSkills --skill evaluating-llms-harness",{"tool":75,"command":76},"bunx","bunx skills add zapabob\u002FSkills --skill evaluating-llms-harness",{"tool":78,"command":79},"pnpm","pnpm dlx skills add zapabob\u002FSkills --skill evaluating-llms-harness","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002F41\u002F41e2dd15ab0d1744355ed1e5763d3f396ba5efbcd3a179c745e85c3e364aefb0\u002FSKILL.md",true,[],[84,122,157,181,208,225],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":103,"publishedAt":39},"01M6GJTE1GESGGF9K028Y7753E","xurl","X\u002FTwitter via xurl CLI: post, search, DM, media, v2 API.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fxurl\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[90,95,98],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":96,"categoryName":97,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[86,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,61,119,120,121],"openclaw","macos","cli","github","linux","windows","official","search","https","social-media","api","Media","hermes","hermes agent","twitter","social","via","post",{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":139,"publishedAt":39},"01M6GJT8S1PXQ8WPXT1HSNBWH2","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwriting-plans\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[128,131,134,135,136],{"categoryNo":129,"categoryName":130,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":132,"categoryName":133,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":96,"categoryName":97,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":137,"categoryName":138,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000036","内容创作者",[140,141,142,143,105,144,108,145,109,146,147,148,149,150,151,152,153,154,155,156],"writing","writing plans","plans","design","review","workflow","documentation","tasks","development","code","planning","code-review","subagent","write","superpowers","paths","test-driven-development",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":166,"publishedAt":39},"01M6GJST0H5X7WHBJ3BZK0DVKH","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwhisper\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[163],{"categoryNo":164,"categoryName":165,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000016","图像与多媒体",[159,105,108,167,168,169,22,170,171,172,30,173,26,174,175,176,177,178,179,180],"openai","speech","multimodal","ASR","Audio","text","speech recognition","general","multilingual","translation","english","identification","processing","Audio Processing",{"skillNo":182,"name":183,"nameZh":6,"description":184,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":190,"publishedAt":39},"01M6GJSMRQ8EETZTYMG72P70EA","weights-and-biases","W&B: log ML experiments, sweeps, model registry, dashboards.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fweights-and-biases\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[187],{"categoryNo":188,"categoryName":189,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000013","分析与可视化",[191,105,192,193,109,194,195,196,197,198,199,30,200,201,202,203,204,205,206,207],"weights and biases","tensorflow","experiments","pytorch","visualization","registry","real-time","tuning","dashboards","experiment","log","time","Experiment Tracking","Model Registry","Hyperparameter Tuning","WandB","Real-Time Visualization",{"skillNo":209,"name":210,"nameZh":6,"description":211,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":212,"categories":213,"keywords":215,"publishedAt":39},"01M6GJS19KDPRY112JTR326S06","watchers","Poll RSS, JSON APIs, and GitHub with watermark dedup.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwatchers\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[214],{"categoryNo":96,"categoryName":97,"taxonomyType":93,"taxonomyTypeDesc":94},[210,105,216,107,108,217,218,219,220,221,222,116,117,223,224],"terminal","automation","http","monitoring","cron","json","rss","devops","with",{"skillNo":226,"name":227,"nameZh":6,"description":228,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":229,"categories":230,"keywords":233,"publishedAt":39},"01M6GJRHJD2A6Q830SWF9W48BC","serving-llms-vllm","vLLM: high-throughput LLM serving, OpenAI API, quantization.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fvllm\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[231,232],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},[21,31,234,108,38,167,235,236,237,22,238,114,239,240,241,242,243,244,245,246],"serving llms vllm","openai api","tensor","Inference Serving","Production","llm serving","High Throughput","PagedAttention","Continuous Batching","Quantization","Tensor Parallelism","high-throughput","throughput"]