[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KPZGE0FTY3X2EARMB0ECCC":3,"skill-same:01M6KPZGE0FTY3X2EARMB0ECCC":93},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":55,"commitSha":56,"skillDirectory":57,"forksCount":13,"repositoryHtmlUrl":58,"repositoryHomepageUrl":6,"ownerHtmlUrl":59,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":60,"install":61,"resources":74},"01M6KPZGE0FTY3X2EARMB0ECCC","evaluating-llms-harness",null,"Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"harness","evaluating llms harness","llms","model","when","orchestra","EleutherAI","GSM8K","LM Evaluation Harness","progress","across","use","eval","Transformers","vLLM","MMLU","HumanEval","Evaluation","reporting","training","benchmarking","2026-11-10 10:12:30",false,"---\nname: evaluating-llms-harness\ndescription: Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard]\ndependencies: [lm-eval, transformers, vllm]\n---\n\n# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\n**Issue: Evaluation too slow**\n\nUse vLLM backend:\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=model-name,tensor_parallel_size=2\n```\n\nOr reduce fewshot examples:\n```bash\n--num_fewshot 0  # Instead of 5\n```\n\nOr evaluate subset of MMLU:\n```bash\n--tasks mmlu_stem  # Only STEM subjects\n```\n\n**Issue: Out of memory**\n\nReduce batch size:\n```bash\n--batch_size 1  # Or --batch_size auto\n```\n\nUse quantization:\n```bash\n--model_args pretrained=model-name,load_in_8bit=True\n```\n\nEnable CPU offloading:\n```bash\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\n```\n\n**Issue: Different results than reported**\n\nCheck fewshot count:\n```bash\n--num_fewshot 5  # Most papers use 5-shot\n```\n\nCheck exact task name:\n```bash\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\n```\n\nVerify model and tokenizer match:\n```bash\n--model_args pretrained=model-name,tokenizer=same-model-name\n```\n\n**Issue: HumanEval not executing code**\n\nInstall execution dependencies:\n```bash\npip install human-eval\n```\n\nEnable code execution:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=model-name \\\n  --tasks humaneval \\\n  --allow_code_execution  # Required for HumanEval\n```\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\n- **VRAM**:\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\n  - 70B model: Requires multi-GPU or quantization\n- **Time** (7B model, single A100):\n  - HellaSwag: 10 minutes\n  - GSM8K: 5 minutes\n  - MMLU (full): 2 hours\n  - HumanEval: 20 minutes\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n\n\n","# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\n**Issue: Evaluation too slow**\n\nUse vLLM backend:\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=model-name,tensor_parallel_size=2\n```\n\nOr reduce fewshot examples:\n```bash\n--num_fewshot 0  # Instead of 5\n```\n\nOr evaluate subset of MMLU:\n```bash\n--tasks mmlu_stem  # Only STEM subjects\n```\n\n**Issue: Out of memory**\n\nReduce batch size:\n```bash\n--batch_size 1  # Or --batch_size auto\n```\n\nUse quantization:\n```bash\n--model_args pretrained=model-name,load_in_8bit=True\n```\n\nEnable CPU offloading:\n```bash\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\n```\n\n**Issue: Different results than reported**\n\nCheck fewshot count:\n```bash\n--num_fewshot 5  # Most papers use 5-shot\n```\n\nCheck exact task name:\n```bash\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\n```\n\nVerify model and tokenizer match:\n```bash\n--model_args pretrained=model-name,tokenizer=same-model-name\n```\n\n**Issue: HumanEval not executing code**\n\nInstall execution dependencies:\n```bash\npip install human-eval\n```\n\nEnable code execution:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=model-name \\\n  --tasks humaneval \\\n  --allow_code_execution  # Required for HumanEval\n```\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\n- **VRAM**:\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\n  - 70B model: Requires multi-GPU or quantization\n- **Time** (7B model, single A100):\n  - HellaSwag: 10 minutes\n  - GSM8K: 5 minutes\n  - MMLU (full): 2 hours\n  - HumanEval: 20 minutes\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n\n\n",{"name":5,"tags":44,"author":49,"license":14,"version":50,"description":7,"dependencies":51},[35,26,45,33,34,25,24,46,47,48],"Benchmarking","Model Quality","Academic Benchmarks","Industry Standard","Orchestra Research","1.0.0",[52,53,54],"lm-eval","transformers","vllm","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-12 18:13:37",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add endman100\u002Fskill-router-nexus --skill evaluating-llms-harness",{"tool":67,"command":68},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill evaluating-llms-harness",{"tool":70,"command":71},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill evaluating-llms-harness","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness\u002FSKILL.md",true,[75,81,85,89],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fapi-evaluation.md","REFERENCE","参考文档",11114,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness\u002Freferences\u002Fapi-evaluation.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fbenchmark-guide.md",10769,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness\u002Freferences\u002Fbenchmark-guide.md",{"path":86,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fcustom-tasks.md",13125,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness\u002Freferences\u002Fcustom-tasks.md",{"path":90,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Fdistributed-eval.md",11426,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Evaluation-and-Benchmarking\u002Flm-evaluation-harness\u002Flm-evaluation-harness\u002Freferences\u002Fdistributed-eval.md",[94,118,155,177,186,210],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":98,"sourceHtmlUrl":99,"categories":100,"keywords":106,"publishedAt":39},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[101],{"categoryNo":102,"categoryName":103,"taxonomyType":104,"taxonomyTypeDesc":105},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[107,108,109,110,111,112,113,114,115,116,117],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":122,"sourceHtmlUrl":123,"categories":124,"keywords":136,"publishedAt":39},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[125,130,133],{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":131,"categoryName":132,"taxonomyType":128,"taxonomyTypeDesc":129},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":134,"categoryName":135,"taxonomyType":104,"taxonomyTypeDesc":105},"01JBSEEDCAT000000000000036","内容创作者",[137,138,139,140,141,142,143,144,145,146,29,147,148,110,149,150,151,152,153,154],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","style","distill","analysis","frame","source","for","at","complete",{"skillNo":156,"name":157,"nameZh":6,"description":158,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":159,"sourceHtmlUrl":160,"categories":161,"keywords":164,"publishedAt":39},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[162,163],{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},{"categoryNo":134,"categoryName":135,"taxonomyType":104,"taxonomyTypeDesc":105},[165,166,167,168,22,169,170,171,172,29,173,174,175,152,176,139],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":178,"name":179,"nameZh":6,"description":180,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":181,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":39},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[179,185],"文本",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":190,"sourceHtmlUrl":191,"categories":192,"keywords":193,"publishedAt":39},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[188,194,195,196,197,198,29,199,115,151,200,201,202,203,117,204,205,206,207,22,208,209,165],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":211,"name":212,"nameZh":6,"description":213,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":214,"sourceHtmlUrl":215,"categories":216,"keywords":218,"publishedAt":39},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[217],{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},[219,220,221,115,114,113,139,110,185,222,165,167],"humanizer","zh","humanizer zh","翻译"]