[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MA1RCXGSA93XTM04ZS7Y5F":3,"skill-same:01M6MA1RCXGSA93XTM04ZS7Y5F":101},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":58,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6MA1RCXGSA93XTM04ZS7Y5F","evaluating-llms-harness",null,"Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1571,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"harness","evaluating llms harness","llms","Transformers","when","GSM8K","LM Evaluation Harness","progress","across","use","eval","model","Safety","vLLM","MMLU","HumanEval","Evaluation","reporting","training","ml","benchmarking","2026-11-10 10:12:30",false,"---\nname: evaluating-llms-harness\ndescription: Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-evaluation-and-safety\n  maintainer: Kalaris Labs\n  tags: Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard\n  dependencies: lm-eval, transformers, vllm\n---\n\n# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\nDetails, code examples and parameter tables: [references\u002Fcommon-issues.md](references\u002Fcommon-issues.md). Read it when this step applies.\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\nDetails, code examples and parameter tables: [references\u002Fhardware-requirements.md](references\u002Fhardware-requirements.md). Read it when this step applies.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm the model, checkpoint, benchmark version and evaluation harness version.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Evaluate on a small subset to verify prompts, parsing and scoring.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| Scores differ from published numbers | Check prompt format, few-shot settings and harness version before concluding. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `evaluating-code-models`: Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics.\n- `nemo-evaluator-sdk`: Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n","# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\nDetails, code examples and parameter tables: [references\u002Fcommon-issues.md](references\u002Fcommon-issues.md). Read it when this step applies.\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\nDetails, code examples and parameter tables: [references\u002Fhardware-requirements.md](references\u002Fhardware-requirements.md). Read it when this step applies.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm the model, checkpoint, benchmark version and evaluation harness version.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Evaluate on a small subset to verify prompts, parsing and scoring.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| Scores differ from published numbers | Check prompt format, few-shot settings and harness version before concluding. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `evaluating-code-models`: Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics.\n- `nemo-evaluator-sdk`: Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"dependencies":50},"Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard","1.0.0","ml-evaluation-and-safety","Kalaris Labs","lm-eval, transformers, vllm","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness",490,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 02:12:02",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill evaluating-llms-harness",{"tool":67,"command":68},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill evaluating-llms-harness",{"tool":70,"command":71},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill evaluating-llms-harness","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002FSKILL.md",true,[75,81,85,89,93,97],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fapi-evaluation.md","REFERENCE","参考文档",11114,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002Freferences\u002Fapi-evaluation.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fbenchmark-guide.md",10769,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002Freferences\u002Fbenchmark-guide.md",{"path":86,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fcommon-issues.md",1197,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002Freferences\u002Fcommon-issues.md",{"path":90,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Fcustom-tasks.md",13125,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002Freferences\u002Fcustom-tasks.md",{"path":94,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":95,"sourceHtmlUrl":96},"references\u002Fdistributed-eval.md",11426,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002Freferences\u002Fdistributed-eval.md",{"path":98,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":99,"sourceHtmlUrl":100},"references\u002Fhardware-requirements.md",362,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fevaluating-llms-harness\u002Freferences\u002Fhardware-requirements.md",[102,111,118,125,151,176],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":110},"01M746ZWACVKMZ1BJXJZHYJSSX","migrate-to-e2e","Moves an existing Playwright, Cypress, Selenium, Appium, Detox, Maestro, or TesterArmy suite to e2e one slice at a time, with the old suite running beside the new one until each ported test is proven. Keeps a MIGRATION.md progress file so any later session resumes where the last stopped. Optionally plants a defect to show both suites catch it. Use for \u002Fmigrate-to-e2e, \"migrate our tests to e2e\", \"port these Maestro flows\", \"move our TesterArmy tests\", or when a project has an e2e config and an older suite still running.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fmigrate-to-e2e\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":110},"01M746ZPTAX1Q0FBKAB1CYND7E","e2e","Writes, runs, and debugs end-to-end tests for web and mobile apps with e2e, the agentic test runner, which mixes agent.act goals with exact locator assertions on a Playwright browser or an iOS or Android device. Also drives a project's running app live over the e2e MCP server to look at a screen, check a UI change, take a screenshot or record a video, and try a locator; and runs bug bashes, parallel e2e explore runs with each bug proven by a repro test. Use when a project depends on e2e or has an e2e.config.ts, when asked for end-to-end, browser, mobile, or agentic UI tests, to check a change in the running app, to QA or bug bash an app or branch, or when an e2e run fails.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fe2e\u002FSKILL.md",[],[],{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":110},"01M746ZJN784HVHMDP9MQGCSGG","create-verification-skill","Generate a project-local verify-\u003Capp> skill on top of e2e, so any coding agent can launch the app, check it is worth driving, exercise a feature the way a user does, keep evidence, and bug bash it. Writes the skill, a feature map with one explore charter per feature, and proves it once. Use for \u002Fcreate-verification-skill, \"make a verification skill for this repo\", or when a project has no scripted way to prove UI behavior.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fcreate-verification-skill\u002FSKILL.md",[],[],{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":131,"publishedAt":40},"01M716A6ETNB9B1SQ3GXZC75VR","deja-search","deja-vu memory — search the user's past AI coding sessions with the deja CLI. Holds what they already settled and never wrote into the repo (naming rules, banned or already-tried tools, required checks before merge, preferred values, who owns what). Use before your first edit in a task, before you call a change done or ready to merge, before you recommend a tool, library, design or reviewer, before re-debugging an error that may already be solved, when they say 'didn't we fix this before', 'what did we decide about X' or 'I already have that', and before telling them something on this machine does not exist.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-search\u002FSKILL.md",[],[132,133,134,135,136,137,138,139,140,141,142,23,143,144,145,146,147,148,149,150],"search","deja search","never","error","ready","memory","re","decide","fix","design","debugging","you","done","reviewer","io","change","cli","merge","past",{"skillNo":152,"name":153,"nameZh":6,"description":154,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":155,"categories":156,"keywords":157,"publishedAt":40},"01M716A3J2W5SENSE9VGMXB375","deja-reflect","End-of-session reflection and memory curation with deja-vu. Review what was decided, what commands broke and what repaired them, and record durable project rules or decisions with deja remember so the next session starts with them.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-reflect\u002FSKILL.md",[],[158,159,160,146,161,162,163,164,165,166,167,168,169,170,137,171,172,173,174,175],"deja reflect","reflect","review","with","record","github","next","https","node","Rules","decisions","brew","remember","project rules","project","session","end","reflection",{"skillNo":177,"name":178,"nameZh":6,"description":179,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":180,"categories":181,"keywords":193,"publishedAt":40},"01M715T3RR4BAYSR57HPPTJ0PF","revid","Make, edit, export and post AI videos with Revid (revid.ai) through the Revid MCP server. Use whenever the user wants a short-form or long-form video made (from an idea, a script, a URL, an article, a PDF, a tweet, a product page, their own clips, or a song), wants to find what is going viral on TikTok, YouTube, Instagram or X before making something, wants to edit or re-export an existing Revid project, generate a standalone AI image or clip, make a series episode, or publish and schedule a finished video to connected social accounts. Covers setup (hosted MCP with OAuth, or an API key), the 36 tools, credits, polling and the rules for publishing. Routes to the more specific revid-* skills when one fits.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftmakersas\u002Frevid-skills\u002Fskills\u002Frevid\u002FSKILL.md",[182,187,190],{"categoryNo":183,"categoryName":184,"taxonomyType":185,"taxonomyTypeDesc":186},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":188,"categoryName":189,"taxonomyType":185,"taxonomyTypeDesc":186},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":191,"categoryName":192,"taxonomyType":185,"taxonomyTypeDesc":186},"01JBSEEDCAT000000000000010","后端与接口",[178,194,195,196,197,198,199,172,200,138,201,202,203,204,23,205,206,207,208,209],"URL","Video","short-form","standalone","PDF","viral","tiktok","short","social","MCP","song","image","mcp server","schedule","api","publish"]