[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EB0JGCFW9KBVBZXA9ECWR1":3,"skill-same:01M6EB0JGCFW9KBVBZXA9ECWR1":166},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":54,"commitSha":55,"skillDirectory":56,"forksCount":57,"repositoryHtmlUrl":58,"repositoryHomepageUrl":6,"ownerHtmlUrl":59,"ownerBlogUrl":6,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6EB0JGCFW9KBVBZXA9ECWR1","evaluating-llms-harness",null,"Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.","luokai0\u002Fai-agent-skills-by-luo-kai","luokai0","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F266954973?v=4","01M6E16VNT42JB8B3X3254QWJ7","luo kai","01M6E16VTFVP6S5XJF1V1MPV90",14,"MIT","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"llms","harness","evaluating llms harness","Industry Standard","Academic Benchmarks","Model Quality","EleutherAI","GSM8K","LM Evaluation Harness","progress","across","benchmarks","use","eval","model","vLLM","HuggingFace","MMLU","training","benchmarking","2026-11-10 10:12:30",false,"---\nname: evaluating-llms-harness\ndescription: Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.\nversion: 1.0.0\nauthor: luo-kai\nlicense: MIT\ntags: [Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard]\ndependencies: [lm-eval, transformers, vllm]\n---\n\n# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\n**Issue: Evaluation too slow**\n\nUse vLLM backend:\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=model-name,tensor_parallel_size=2\n```\n\nOr reduce fewshot examples:\n```bash\n--num_fewshot 0  # Instead of 5\n```\n\nOr evaluate subset of MMLU:\n```bash\n--tasks mmlu_stem  # Only STEM subjects\n```\n\n**Issue: Out of memory**\n\nReduce batch size:\n```bash\n--batch_size 1  # Or --batch_size auto\n```\n\nUse quantization:\n```bash\n--model_args pretrained=model-name,load_in_8bit=True\n```\n\nEnable CPU offloading:\n```bash\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\n```\n\n**Issue: Different results than reported**\n\nCheck fewshot count:\n```bash\n--num_fewshot 5  # Most papers use 5-shot\n```\n\nCheck exact task name:\n```bash\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\n```\n\nVerify model and tokenizer match:\n```bash\n--model_args pretrained=model-name,tokenizer=same-model-name\n```\n\n**Issue: HumanEval not executing code**\n\nInstall execution dependencies:\n```bash\npip install human-eval\n```\n\nEnable code execution:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=model-name \\\n  --tasks humaneval \\\n  --allow_code_execution  # Required for HumanEval\n```\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\n- **VRAM**:\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\n  - 70B model: Requires multi-GPU or quantization\n- **Time** (7B model, single A100):\n  - HellaSwag: 10 minutes\n  - GSM8K: 5 minutes\n  - MMLU (full): 2 hours\n  - HumanEval: 20 minutes\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n\n\n","# lm-evaluation-harness - LLM Benchmarking\n\n## Quick start\n\nlm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.\n\n**Installation**:\n```bash\npip install lm-eval\n```\n\n**Evaluate any HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag \\\n  --device cuda:0 \\\n  --batch_size 8\n```\n\n**View available tasks**:\n```bash\nlm_eval --tasks list\n```\n\n## Common workflows\n\n### Workflow 1: Standard benchmark evaluation\n\nEvaluate model on core benchmarks (MMLU, GSM8K, HumanEval).\n\nCopy this checklist:\n\n```\nBenchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Analyze results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Core reasoning benchmarks**:\n- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice\n- **GSM8K** - Grade school math word problems\n- **HellaSwag** - Common sense reasoning\n- **TruthfulQA** - Truthfulness and factuality\n- **ARC** (AI2 Reasoning Challenge) - Science questions\n\n**Code benchmarks**:\n- **HumanEval** - Python code generation (164 problems)\n- **MBPP** (Mostly Basic Python Problems) - Python coding\n\n**Standard suite** (recommended for model releases):\n```bash\n--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge\n```\n\n**Step 2: Configure model**\n\n**HuggingFace model**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,dtype=bfloat16 \\\n  --tasks mmlu \\\n  --device cuda:0 \\\n  --batch_size auto  # Auto-detect optimal batch size\n```\n\n**Quantized model (4-bit\u002F8-bit)**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,load_in_4bit=True \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Custom checkpoint**:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=\u002Fpath\u002Fto\u002Fmy-model,tokenizer=\u002Fpath\u002Fto\u002Ftokenizer \\\n  --tasks mmlu \\\n  --device cuda:0\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full MMLU evaluation (57 subjects)\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --num_fewshot 5 \\  # 5-shot evaluation (standard)\n  --batch_size 8 \\\n  --output_path results\u002F \\\n  --log_samples  # Save individual predictions\n\n# Multiple benchmarks at once\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \\\n  --num_fewshot 5 \\\n  --batch_size 8 \\\n  --output_path results\u002Fllama2-7b-eval.json\n```\n\n**Step 4: Analyze results**\n\nResults saved to `results\u002Fllama2-7b-eval.json`:\n\n```json\n{\n  \"results\": {\n    \"mmlu\": {\n      \"acc\": 0.459,\n      \"acc_stderr\": 0.004\n    },\n    \"gsm8k\": {\n      \"exact_match\": 0.142,\n      \"exact_match_stderr\": 0.006\n    },\n    \"hellaswag\": {\n      \"acc_norm\": 0.765,\n      \"acc_norm_stderr\": 0.004\n    }\n  },\n  \"config\": {\n    \"model\": \"hf\",\n    \"model_args\": \"pretrained=meta-llama\u002FLlama-2-7b-hf\",\n    \"num_fewshot\": 5\n  }\n}\n```\n\n### Workflow 2: Track training progress\n\nEvaluate checkpoints during training.\n\n```\nTraining Progress Tracking:\n- [ ] Step 1: Set up periodic evaluation\n- [ ] Step 2: Choose quick benchmarks\n- [ ] Step 3: Automate evaluation\n- [ ] Step 4: Plot learning curves\n```\n\n**Step 1: Set up periodic evaluation**\n\nEvaluate every N training steps:\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_checkpoint.sh\n\nCHECKPOINT_DIR=$1\nSTEP=$2\n\nlm_eval --model hf \\\n  --model_args pretrained=$CHECKPOINT_DIR\u002Fcheckpoint-$STEP \\\n  --tasks gsm8k,hellaswag \\\n  --num_fewshot 0 \\  # 0-shot for speed\n  --batch_size 16 \\\n  --output_path results\u002Fstep-$STEP.json\n```\n\n**Step 2: Choose quick benchmarks**\n\nFast benchmarks for frequent evaluation:\n- **HellaSwag**: ~10 minutes on 1 GPU\n- **GSM8K**: ~5 minutes\n- **PIQA**: ~2 minutes\n\nAvoid for frequent eval (too slow):\n- **MMLU**: ~2 hours (57 subjects)\n- **HumanEval**: Requires code execution\n\n**Step 3: Automate evaluation**\n\nIntegrate with training script:\n\n```python\n# In training loop\nif step % eval_interval == 0:\n    model.save_pretrained(f\"checkpoints\u002Fstep-{step}\")\n\n    # Run evaluation\n    os.system(f\".\u002Feval_checkpoint.sh checkpoints step-{step}\")\n```\n\nOr use PyTorch Lightning callbacks:\n\n```python\nfrom pytorch_lightning import Callback\n\nclass EvalHarnessCallback(Callback):\n    def on_validation_epoch_end(self, trainer, pl_module):\n        step = trainer.global_step\n        checkpoint_path = f\"checkpoints\u002Fstep-{step}\"\n\n        # Save checkpoint\n        trainer.save_checkpoint(checkpoint_path)\n\n        # Run lm-eval\n        os.system(f\"lm_eval --model hf --model_args pretrained={checkpoint_path} ...\")\n```\n\n**Step 4: Plot learning curves**\n\n```python\nimport json\nimport matplotlib.pyplot as plt\n\n# Load all results\nsteps = []\nmmlu_scores = []\n\nfor file in sorted(glob.glob(\"results\u002Fstep-*.json\")):\n    with open(file) as f:\n        data = json.load(f)\n        step = int(file.split(\"-\")[1].split(\".\")[0])\n        steps.append(step)\n        mmlu_scores.append(data[\"results\"][\"mmlu\"][\"acc\"])\n\n# Plot\nplt.plot(steps, mmlu_scores)\nplt.xlabel(\"Training Step\")\nplt.ylabel(\"MMLU Accuracy\")\nplt.title(\"Training Progress\")\nplt.savefig(\"training_curve.png\")\n```\n\n### Workflow 3: Compare multiple models\n\nBenchmark suite for model comparison.\n\n```\nModel Comparison:\n- [ ] Step 1: Define model list\n- [ ] Step 2: Run evaluations\n- [ ] Step 3: Generate comparison table\n```\n\n**Step 1: Define model list**\n\n```bash\n# models.txt\nmeta-llama\u002FLlama-2-7b-hf\nmeta-llama\u002FLlama-2-13b-hf\nmistralai\u002FMistral-7B-v0.1\nmicrosoft\u002Fphi-2\n```\n\n**Step 2: Run evaluations**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_all_models.sh\n\nTASKS=\"mmlu,gsm8k,hellaswag,truthfulqa\"\n\nwhile read model; do\n    echo \"Evaluating $model\"\n\n    # Extract model name for output file\n    model_name=$(echo $model | sed 's\u002F\\\u002F\u002F-\u002Fg')\n\n    lm_eval --model hf \\\n      --model_args pretrained=$model,dtype=bfloat16 \\\n      --tasks $TASKS \\\n      --num_fewshot 5 \\\n      --batch_size auto \\\n      --output_path results\u002F$model_name.json\n\ndone \u003C models.txt\n```\n\n**Step 3: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\n    \"meta-llama-Llama-2-7b-hf\",\n    \"meta-llama-Llama-2-13b-hf\",\n    \"mistralai-Mistral-7B-v0.1\",\n    \"microsoft-phi-2\"\n]\n\ntasks = [\"mmlu\", \"gsm8k\", \"hellaswag\", \"truthfulqa\"]\n\nresults = []\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        row = {\"Model\": model.replace(\"-\", \"\u002F\")}\n        for task in tasks:\n            # Get primary metric for each task\n            metrics = data[\"results\"][task]\n            if \"acc\" in metrics:\n                row[task.upper()] = f\"{metrics['acc']:.3f}\"\n            elif \"exact_match\" in metrics:\n                row[task.upper()] = f\"{metrics['exact_match']:.3f}\"\n        results.append(row)\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\nOutput:\n```\n| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |\n|------------------------|-------|-------|-----------|------------|\n| meta-llama\u002FLlama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |\n| meta-llama\u002FLlama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |\n| mistralai\u002FMistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |\n| microsoft\u002Fphi-2        | 0.560 | 0.613 | 0.682     | 0.447      |\n```\n\n### Workflow 4: Evaluate with vLLM (faster inference)\n\nUse vLLM backend for 5-10x faster evaluation.\n\n```\nvLLM Evaluation:\n- [ ] Step 1: Install vLLM\n- [ ] Step 2: Configure vLLM backend\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Install vLLM**\n\n```bash\npip install vllm\n```\n\n**Step 2: Configure vLLM backend**\n\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n**Step 3: Run evaluation**\n\nvLLM is 5-10× faster than standard HuggingFace:\n\n```bash\n# Standard HF: ~2 hours for MMLU on 7B model\nlm_eval --model hf \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf \\\n  --tasks mmlu \\\n  --batch_size 8\n\n# vLLM: ~15-20 minutes for MMLU on 7B model\nlm_eval --model vllm \\\n  --model_args pretrained=meta-llama\u002FLlama-2-7b-hf,tensor_parallel_size=2 \\\n  --tasks mmlu \\\n  --batch_size auto\n```\n\n## When to use vs alternatives\n\n**Use lm-evaluation-harness when:**\n- Benchmarking models for academic papers\n- Comparing model quality across standard tasks\n- Tracking training progress\n- Reporting standardized metrics (everyone uses same prompts)\n- Need reproducible evaluation\n\n**Use alternatives instead:**\n- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)\n- **AlpacaEval**: Instruction-following evaluation with LLM judges\n- **MT-Bench**: Conversational multi-turn evaluation\n- **Custom scripts**: Domain-specific evaluation\n\n## Common issues\n\n**Issue: Evaluation too slow**\n\nUse vLLM backend:\n```bash\nlm_eval --model vllm \\\n  --model_args pretrained=model-name,tensor_parallel_size=2\n```\n\nOr reduce fewshot examples:\n```bash\n--num_fewshot 0  # Instead of 5\n```\n\nOr evaluate subset of MMLU:\n```bash\n--tasks mmlu_stem  # Only STEM subjects\n```\n\n**Issue: Out of memory**\n\nReduce batch size:\n```bash\n--batch_size 1  # Or --batch_size auto\n```\n\nUse quantization:\n```bash\n--model_args pretrained=model-name,load_in_8bit=True\n```\n\nEnable CPU offloading:\n```bash\n--model_args pretrained=model-name,device_map=auto,offload_folder=offload\n```\n\n**Issue: Different results than reported**\n\nCheck fewshot count:\n```bash\n--num_fewshot 5  # Most papers use 5-shot\n```\n\nCheck exact task name:\n```bash\n--tasks mmlu  # Not mmlu_direct or mmlu_fewshot\n```\n\nVerify model and tokenizer match:\n```bash\n--model_args pretrained=model-name,tokenizer=same-model-name\n```\n\n**Issue: HumanEval not executing code**\n\nInstall execution dependencies:\n```bash\npip install human-eval\n```\n\nEnable code execution:\n```bash\nlm_eval --model hf \\\n  --model_args pretrained=model-name \\\n  --tasks humaneval \\\n  --allow_code_execution  # Required for HumanEval\n```\n\n## Advanced topics\n\n**Benchmark descriptions**: See [references\u002Fbenchmark-guide.md](references\u002Fbenchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation.\n\n**Custom tasks**: See [references\u002Fcustom-tasks.md](references\u002Fcustom-tasks.md) for creating domain-specific evaluation tasks.\n\n**API evaluation**: See [references\u002Fapi-evaluation.md](references\u002Fapi-evaluation.md) for evaluating OpenAI, Anthropic, and other API models.\n\n**Multi-GPU strategies**: See [references\u002Fdistributed-eval.md](references\u002Fdistributed-eval.md) for data parallel and tensor parallel evaluation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow)\n- **VRAM**:\n  - 7B model: 16GB (bf16) or 8GB (8-bit)\n  - 13B model: 28GB (bf16) or 14GB (8-bit)\n  - 70B model: Requires multi-GPU or quantization\n- **Time** (7B model, single A100):\n  - HellaSwag: 10 minutes\n  - GSM8K: 5 minutes\n  - MMLU (full): 2 hours\n  - HumanEval: 20 minutes\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\n- Docs: https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.\n- Leaderboard: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FHuggingFaceH4\u002Fopen_llm_leaderboard (uses this harness)\n\n\n\n",{"name":5,"tags":44,"author":48,"license":15,"version":49,"description":7,"dependencies":50},[45,27,46,36,47,26,25,24,23,22],"Evaluation","Benchmarking","HumanEval","luo-kai","1.0.0",[51,52,53],"lm-eval","transformers","vllm","main","4d2fb89cdfb75f3f6fd3c6c5ec9ad90ea129eb3c","ai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness",1,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai","https:\u002F\u002Fgithub.com\u002Fluokai0","a solo builder.","2026-05-06 04:29:04",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill evaluating-llms-harness",{"tool":68,"command":69},"bunx","bunx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill evaluating-llms-harness",{"tool":71,"command":72},"pnpm","pnpm dlx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill evaluating-llms-harness","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness\u002FSKILL.md",true,[76,82,88,94,98,102,106,110,114,118,122,126,130,134,138,142,146,150,154,158,162],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"customize.toml","OTHER","其他",1614,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fcustomize.toml",{"path":83,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":86,"sourceHtmlUrl":87},"data\u002Fgdd-purpose.md","REFERENCE","参考文档",8922,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fdata\u002Fgdd-purpose.md",{"path":89,"resourceType":90,"resourceTypeDesc":91,"sizeBytes":92,"sourceHtmlUrl":93},"data\u002Fgenre-complexity.csv","ASSET","资产文件",8025,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fdata\u002Fgenre-complexity.csv",{"path":95,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fapi-evaluation.md",11114,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness\u002Freferences\u002Fapi-evaluation.md",{"path":99,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Fbenchmark-guide.md",10769,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness\u002Freferences\u002Fbenchmark-guide.md",{"path":103,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":104,"sourceHtmlUrl":105},"references\u002Fcustom-tasks.md",13125,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness\u002Freferences\u002Fcustom-tasks.md",{"path":107,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":108,"sourceHtmlUrl":109},"references\u002Fdistributed-eval.md",11426,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Flm-evaluation-harness\u002Freferences\u002Fdistributed-eval.md",{"path":111,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":112,"sourceHtmlUrl":113},"steps-v\u002Fstep-v-01-discovery.md",7714,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-01-discovery.md",{"path":115,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":116,"sourceHtmlUrl":117},"steps-v\u002Fstep-v-02-format-detection.md",7015,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-02-format-detection.md",{"path":119,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":120,"sourceHtmlUrl":121},"steps-v\u002Fstep-v-02b-parity-check.md",7156,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-02b-parity-check.md",{"path":123,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":124,"sourceHtmlUrl":125},"steps-v\u002Fstep-v-03-density-validation.md",7348,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-03-density-validation.md",{"path":127,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":128,"sourceHtmlUrl":129},"steps-v\u002Fstep-v-04-brief-coverage-validation.md",8549,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-04-brief-coverage-validation.md",{"path":131,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":132,"sourceHtmlUrl":133},"steps-v\u002Fstep-v-05-measurability-validation.md",9028,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-05-measurability-validation.md",{"path":135,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":136,"sourceHtmlUrl":137},"steps-v\u002Fstep-v-06-traceability-validation.md",8529,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-06-traceability-validation.md",{"path":139,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":140,"sourceHtmlUrl":141},"steps-v\u002Fstep-v-07-implementation-leakage-validation.md",8960,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-07-implementation-leakage-validation.md",{"path":143,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":144,"sourceHtmlUrl":145},"steps-v\u002Fstep-v-08-genre-compliance-validation.md",9500,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-08-genre-compliance-validation.md",{"path":147,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":148,"sourceHtmlUrl":149},"steps-v\u002Fstep-v-09-game-type-validation.md",10861,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-09-game-type-validation.md",{"path":151,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":152,"sourceHtmlUrl":153},"steps-v\u002Fstep-v-10-smart-validation.md",7679,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-10-smart-validation.md",{"path":155,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":156,"sourceHtmlUrl":157},"steps-v\u002Fstep-v-11-holistic-quality-validation.md",9263,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-11-holistic-quality-validation.md",{"path":159,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":160,"sourceHtmlUrl":161},"steps-v\u002Fstep-v-12-completeness-validation.md",10445,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-12-completeness-validation.md",{"path":163,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":164,"sourceHtmlUrl":165},"steps-v\u002Fstep-v-13-report-complete.md",9006,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fgds-validate-gdd\u002Fsteps-v\u002Fstep-v-13-report-complete.md",[167,193,210,240,272,309],{"skillNo":168,"name":169,"nameZh":6,"description":170,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":171,"categories":172,"keywords":173,"publishedAt":39},"01M6F2SKTE208174C7S7JJCZC7","write-contracts","Generates secure Aptos Move V2 smart contracts with Object model, Digital Asset integration, security patterns, and storage type guidance. Includes comprehensive storage decision framework for optimal data structure selection. Triggers on: 'write contract', 'create NFT collection', 'build marketplace', 'implement minting', 'generate Move module', 'create token contract', 'build DAO', 'implement staking'. Ask storage questions when: 'store', 'track', 'registry', 'mapping', 'list', 'collection'.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fwrite-contracts\u002FSKILL.md",[],[174,175,176,177,178,179,180,181,33,182,183,184,185,186,187,188,189,190,191,192],"write","contracts","write contracts","storage","secure","create","generate","collection","decision","ask","smart-contracts","selection","objects","structure","nft","token","aptos","critical","move",{"skillNo":194,"name":195,"nameZh":6,"description":196,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":197,"categories":198,"keywords":199,"publishedAt":39},"01M6F2RFXJ6T9MKW0A7N7GM45B","web-search","通用网络搜索技能，支持多引擎搜索（百度、必应、DuckDuckGo），无需API密钥即可获取实时信息","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fweb-search-ex-skill\u002FSKILL.md",[],[200,201,202,203,204,205,206,207,208,209],"web","search","web search","搜索","baidu","bing","duckduckgo","百度","多引擎","internet",{"skillNo":211,"name":212,"nameZh":6,"description":213,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":214,"categories":215,"keywords":221,"publishedAt":39},"01M6F2K968G1FZEZFPW2E6B222","solo-build","Execute implementation plan tasks with TDD workflow, auto-commit, and phase gates. Use when user says \"build it\", \"start building\", \"execute plan\", \"implement tasks\", \"ship it\", or references a track ID. Do NOT use for planning (use \u002Fplan) or scaffolding (use \u002Fscaffold).","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fsolo-build\u002FSKILL.md",[216],{"categoryNo":217,"categoryName":218,"taxonomyType":219,"taxonomyTypeDesc":220},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[222,223,224,225,226,227,228,229,201,230,231,232,174,233,234,235,236,237,238,239],"build","solo","solo build","MCP","workflow","commit","tasks","code","Bash","plan","read","auto","references","context7","query","scaffolding","ship it","project",{"skillNo":241,"name":242,"nameZh":6,"description":243,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":244,"categories":245,"keywords":252,"publishedAt":39},"01M6F2GKN39SGHSPWYB3662YC8","react-email","Use when building HTML email templates with React components, adding a visual email editor to an application using the React Email visual editor, rendering emails to HTML, or sending emails with Resend. Covers welcome emails, password resets, notifications, order confirmations, newsletters, transactional emails, and the embeddable email editor component.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Freact-email-skills\u002FSKILL.md",[246,249],{"categoryNo":247,"categoryName":248,"taxonomyType":219,"taxonomyTypeDesc":220},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":250,"categoryName":251,"taxonomyType":219,"taxonomyTypeDesc":220},"01JBSEEDCAT000000000000009","前端开发",[253,254,255,256,257,258,259,260,261,262,263,264,265,31,266,267,268,269,270,271],"react","email","react email","github","editor","components","react components","component","welcome","https","notifications","password","node","html","resend","visual","emails","rendering","with",{"skillNo":273,"name":274,"nameZh":6,"description":275,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":276,"categories":277,"keywords":290,"publishedAt":39},"01M6F2FBVGVDFSG8SSFJGXA40C","APITester Agent-Driven API Testing","Test API endpoints and document responses. Define tests in plain English, run them, get formatted results. Agent-driven Postman alternative.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fopenclaw-api-tester\u002FSKILL.md",[278,281,284,285],{"categoryNo":279,"categoryName":280,"taxonomyType":219,"taxonomyTypeDesc":220},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":282,"categoryName":283,"taxonomyType":219,"taxonomyTypeDesc":220},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":217,"categoryName":218,"taxonomyType":219,"taxonomyTypeDesc":220},{"categoryNo":286,"categoryName":287,"taxonomyType":288,"taxonomyTypeDesc":289},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308],"testing","api","api testing","driven","APITester Agent Driven API Testing","test","run","postman","results","plain english","tester","endpoints","document","get","define","tests","english","api-tester",{"skillNo":310,"name":311,"nameZh":6,"description":312,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":313,"categories":314,"keywords":315,"publishedAt":39},"01M6F273Y3NGAPVJZ608BC0G9G","hubspot-implementation-plan","Generate a phased implementation plan from a HubSpot audit report. Creates prioritized, sequenced cleanup processes with effort estimates, dependencies, and automation feasibility. Use after running \u002Fhubspot-audit.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fhubspot-implementation-plan\u002FSKILL.md",[],[231,316,317,318,319,320,321,180,322,323,324,325,326,31,327,328,271,329],"implementation","hubspot","hubspot implementation plan","automation","feasibility","cleanup","planning","report","audit","audit report","phased implementation","dependencies","processes","hubspot audit"]