[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CKDFZWGK7RMAM57YW6FZJC":3,"skill-same:01M7CKDFZWGK7RMAM57YW6FZJC":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":44,"repositoryHtmlUrl":45,"repositoryHomepageUrl":6,"ownerHtmlUrl":46,"ownerBlogUrl":6,"ownerDescription":47,"updatedAt":48,"install":49,"resources":62},"01M7CKDFZWGK7RMAM57YW6FZJC","evaluating-code-models",null,"Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.","Carchaloneo22\u002FClaude-Code-template","Carchaloneo22","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F159488757?v=4","01M7CK5MQXAFZE3W7SAP69KNYR","Carlos Chaparro","01M7CK5MRFE26CDPEQDN5RSQPK",2,"MIT","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fevaluation-bigcode-evaluation-harness\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: evaluating-code-models\ndescription: Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Evaluation, Code Generation, HumanEval, MBPP, MultiPL-E, Pass@k, BigCode, Benchmarking, Code Models]\ndependencies: [bigcode-evaluation-harness, transformers>=4.25.1, accelerate>=0.13.2, datasets>=2.6.1]\n---\n\n# BigCode Evaluation Harness - Code Model Benchmarking\n\n## Quick Start\n\nBigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).\n\n**Installation**:\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness.git\ncd bigcode-evaluation-harness\npip install -e .\naccelerate config\n```\n\n**Evaluate on HumanEval**:\n```bash\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --n_samples 20 \\\n  --batch_size 10 \\\n  --allow_code_execution \\\n  --save_generations\n```\n\n**View available tasks**:\n```bash\npython -c \"from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)\"\n```\n\n## Common Workflows\n\n### Workflow 1: Standard Code Benchmark Evaluation\n\nEvaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).\n\n**Checklist**:\n```\nCode Benchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model and generation\n- [ ] Step 3: Run evaluation with code execution\n- [ ] Step 4: Analyze pass@k results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Python code generation** (most common):\n- **HumanEval**: 164 handwritten problems, function completion\n- **HumanEval+**: Same 164 problems with 80× more tests (stricter)\n- **MBPP**: 500 crowd-sourced problems, entry-level difficulty\n- **MBPP+**: 399 curated problems with 35× more tests\n\n**Multi-language** (18 languages):\n- **MultiPL-E**: HumanEval\u002FMBPP translated to C++, Java, JavaScript, Go, Rust, etc.\n\n**Advanced**:\n- **APPS**: 10,000 problems (introductory\u002Finterview\u002Fcompetition)\n- **DS-1000**: 1,000 data science problems across 7 libraries\n\n**Step 2: Configure model and generation**\n\n```bash\n# Standard HuggingFace model\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --do_sample True \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution\n\n# Quantized model (4-bit)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-34b-hf \\\n  --tasks humaneval \\\n  --load_in_4bit \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n\n# Custom\u002Fprivate model\naccelerate launch main.py \\\n  --model \u002Fpath\u002Fto\u002Fmy-code-model \\\n  --tasks humaneval \\\n  --trust_remote_code \\\n  --use_auth_token \\\n  --allow_code_execution\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full evaluation with pass@k estimation (k=1,10,100)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --temperature 0.8 \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution \\\n  --save_generations \\\n  --metric_output_path results\u002Fstarcoder2-humaneval.json\n```\n\n**Step 4: Analyze results**\n\nResults in `results\u002Fstarcoder2-humaneval.json`:\n```json\n{\n  \"humaneval\": {\n    \"pass@1\": 0.354,\n    \"pass@10\": 0.521,\n    \"pass@100\": 0.689\n  },\n  \"config\": {\n    \"model\": \"bigcode\u002Fstarcoder2-7b\",\n    \"temperature\": 0.8,\n    \"n_samples\": 200\n  }\n}\n```\n\n### Workflow 2: Multi-Language Evaluation (MultiPL-E)\n\nEvaluate code generation across 18 programming languages.\n\n**Checklist**:\n```\nMulti-Language Evaluation:\n- [ ] Step 1: Generate solutions (host machine)\n- [ ] Step 2: Run evaluation in Docker (safe execution)\n- [ ] Step 3: Compare across languages\n```\n\n**Step 1: Generate solutions on host**\n\n```bash\n# Generate without execution (safe)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --max_length_generation 650 \\\n  --temperature 0.8 \\\n  --n_samples 50 \\\n  --batch_size 50 \\\n  --generation_only \\\n  --save_generations \\\n  --save_generations_path generations_multi.json\n```\n\n**Step 2: Evaluate in Docker container**\n\n```bash\n# Pull the MultiPL-E Docker image\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n\n# Run evaluation inside container\ndocker run -v $(pwd)\u002Fgenerations_multi.json:\u002Fapp\u002Fgenerations.json:ro \\\n  -it evaluation-harness-multiple python3 main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --load_generations_path \u002Fapp\u002Fgenerations.json \\\n  --allow_code_execution \\\n  --n_samples 50\n```\n\n**Supported languages**: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket\n\n### Workflow 3: Instruction-Tuned Model Evaluation\n\nEvaluate chat\u002Finstruction models with proper formatting.\n\n**Checklist**:\n```\nInstruction Model Evaluation:\n- [ ] Step 1: Use instruction-tuned tasks\n- [ ] Step 2: Configure instruction tokens\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Choose instruction tasks**\n\n- **instruct-humaneval**: HumanEval with instruction prompts\n- **humanevalsynthesize-{lang}**: HumanEvalPack synthesis tasks\n\n**Step 2: Configure instruction tokens**\n\n```bash\n# For models with chat templates (e.g., CodeLlama-Instruct)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks instruct-humaneval \\\n  --instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\" \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n**Step 3: HumanEvalPack for instruction models**\n\n```bash\n# Test code synthesis across 6 languages\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks humanevalsynthesize-python,humanevalsynthesize-js \\\n  --prompt instruct \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n### Workflow 4: Compare Multiple Models\n\nBenchmark suite for model comparison.\n\n**Step 1: Create evaluation script**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_models.sh\n\nMODELS=(\n  \"bigcode\u002Fstarcoder2-7b\"\n  \"codellama\u002FCodeLlama-7b-hf\"\n  \"deepseek-ai\u002Fdeepseek-coder-6.7b-base\"\n)\nTASKS=\"humaneval,mbpp\"\n\nfor model in \"${MODELS[@]}\"; do\n  model_name=$(echo $model | tr '\u002F' '-')\n  echo \"Evaluating $model\"\n\n  accelerate launch main.py \\\n    --model $model \\\n    --tasks $TASKS \\\n    --temperature 0.2 \\\n    --n_samples 20 \\\n    --batch_size 20 \\\n    --allow_code_execution \\\n    --metric_output_path results\u002F${model_name}.json\ndone\n```\n\n**Step 2: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\"bigcode-starcoder2-7b\", \"codellama-CodeLlama-7b-hf\", \"deepseek-ai-deepseek-coder-6.7b-base\"]\nresults = []\n\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        results.append({\n            \"Model\": model,\n            \"HumanEval pass@1\": f\"{data['humaneval']['pass@1']:.3f}\",\n            \"MBPP pass@1\": f\"{data['mbpp']['pass@1']:.3f}\"\n        })\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\n## When to Use vs Alternatives\n\n**Use BigCode Evaluation Harness when:**\n- Evaluating **code generation** models specifically\n- Need **multi-language** evaluation (18 languages via MultiPL-E)\n- Testing **functional correctness** with unit tests (pass@k)\n- Benchmarking for **BigCode\u002FHuggingFace leaderboards**\n- Evaluating **fill-in-the-middle** (FIM) capabilities\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)\n- **EvalPlus**: Stricter HumanEval+\u002FMBPP+ with more test cases\n- **SWE-bench**: Real-world GitHub issue resolution\n- **LiveCodeBench**: Contamination-free, continuously updated problems\n- **CodeXGLUE**: Code understanding tasks (clone detection, defect prediction)\n\n## Supported Benchmarks\n\n| Benchmark | Problems | Languages | Metric | Use Case |\n|-----------|----------|-----------|--------|----------|\n| HumanEval | 164 | Python | pass@k | Standard code completion |\n| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |\n| MBPP | 500 | Python | pass@k | Entry-level problems |\n| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |\n| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |\n| APPS | 10,000 | Python | pass@k | Competition-level |\n| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |\n| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis\u002Ffix\u002Fexplain |\n| Mercury | 1,889 | Python | Efficiency | Computational efficiency |\n\n## Common Issues\n\n**Issue: Different results than reported in papers**\n\nCheck these factors:\n```bash\n# 1. Verify n_samples (need 200 for accurate pass@k)\n--n_samples 200\n\n# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)\n--temperature 0.8\n\n# 3. Verify task name matches exactly\n--tasks humaneval  # Not \"human_eval\" or \"HumanEval\"\n\n# 4. Check max_length_generation\n--max_length_generation 512  # Increase for longer problems\n```\n\n**Issue: CUDA out of memory**\n\n```bash\n# Use quantization\n--load_in_8bit\n# OR\n--load_in_4bit\n\n# Reduce batch size\n--batch_size 1\n\n# Set memory limit\n--max_memory_per_gpu \"20GiB\"\n```\n\n**Issue: Code execution hangs or times out**\n\nUse Docker for safe execution:\n```bash\n# Generate on host (no execution)\n--generation_only --save_generations\n\n# Evaluate in Docker\ndocker run ... --allow_code_execution --load_generations_path ...\n```\n\n**Issue: Low scores on instruction models**\n\nEnsure proper instruction formatting:\n```bash\n# Use instruction-specific tasks\n--tasks instruct-humaneval\n\n# Set instruction tokens for your model\n--instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\"\n```\n\n**Issue: MultiPL-E language failures**\n\nUse the dedicated Docker image:\n```bash\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n```\n\n## Command Reference\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--model` | - | HuggingFace model ID or local path |\n| `--tasks` | - | Comma-separated task names |\n| `--n_samples` | 1 | Samples per problem (200 for pass@k) |\n| `--temperature` | 0.2 | Sampling temperature |\n| `--max_length_generation` | 512 | Max tokens (prompt + generation) |\n| `--batch_size` | 1 | Batch size per GPU |\n| `--allow_code_execution` | False | Enable code execution (required) |\n| `--generation_only` | False | Generate without evaluation |\n| `--load_generations_path` | - | Load pre-generated solutions |\n| `--save_generations` | False | Save generated code |\n| `--metric_output_path` | results.json | Output file for metrics |\n| `--load_in_8bit` | False | 8-bit quantization |\n| `--load_in_4bit` | False | 4-bit quantization |\n| `--trust_remote_code` | False | Allow custom model code |\n| `--precision` | fp32 | Model precision (fp32\u002Ffp16\u002Fbf16) |\n\n## Hardware Requirements\n\n| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |\n|------------|-------------|--------------|-------------------------|\n| 7B | 14GB | 6GB | ~30 min (A100) |\n| 13B | 26GB | 10GB | ~1 hour (A100) |\n| 34B | 68GB | 20GB | ~2 hours (A100) |\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\n- **Documentation**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- **BigCode Leaderboard**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbigcode\u002Fbigcode-models-leaderboard\n- **HumanEval Dataset**: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fopenai_humaneval\n- **MultiPL-E**: https:\u002F\u002Fgithub.com\u002Fnuprl\u002FMultiPL-E\n","# BigCode Evaluation Harness - Code Model Benchmarking\n\n## Quick Start\n\nBigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).\n\n**Installation**:\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness.git\ncd bigcode-evaluation-harness\npip install -e .\naccelerate config\n```\n\n**Evaluate on HumanEval**:\n```bash\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --n_samples 20 \\\n  --batch_size 10 \\\n  --allow_code_execution \\\n  --save_generations\n```\n\n**View available tasks**:\n```bash\npython -c \"from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)\"\n```\n\n## Common Workflows\n\n### Workflow 1: Standard Code Benchmark Evaluation\n\nEvaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).\n\n**Checklist**:\n```\nCode Benchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model and generation\n- [ ] Step 3: Run evaluation with code execution\n- [ ] Step 4: Analyze pass@k results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Python code generation** (most common):\n- **HumanEval**: 164 handwritten problems, function completion\n- **HumanEval+**: Same 164 problems with 80× more tests (stricter)\n- **MBPP**: 500 crowd-sourced problems, entry-level difficulty\n- **MBPP+**: 399 curated problems with 35× more tests\n\n**Multi-language** (18 languages):\n- **MultiPL-E**: HumanEval\u002FMBPP translated to C++, Java, JavaScript, Go, Rust, etc.\n\n**Advanced**:\n- **APPS**: 10,000 problems (introductory\u002Finterview\u002Fcompetition)\n- **DS-1000**: 1,000 data science problems across 7 libraries\n\n**Step 2: Configure model and generation**\n\n```bash\n# Standard HuggingFace model\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --do_sample True \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution\n\n# Quantized model (4-bit)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-34b-hf \\\n  --tasks humaneval \\\n  --load_in_4bit \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n\n# Custom\u002Fprivate model\naccelerate launch main.py \\\n  --model \u002Fpath\u002Fto\u002Fmy-code-model \\\n  --tasks humaneval \\\n  --trust_remote_code \\\n  --use_auth_token \\\n  --allow_code_execution\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full evaluation with pass@k estimation (k=1,10,100)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --temperature 0.8 \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution \\\n  --save_generations \\\n  --metric_output_path results\u002Fstarcoder2-humaneval.json\n```\n\n**Step 4: Analyze results**\n\nResults in `results\u002Fstarcoder2-humaneval.json`:\n```json\n{\n  \"humaneval\": {\n    \"pass@1\": 0.354,\n    \"pass@10\": 0.521,\n    \"pass@100\": 0.689\n  },\n  \"config\": {\n    \"model\": \"bigcode\u002Fstarcoder2-7b\",\n    \"temperature\": 0.8,\n    \"n_samples\": 200\n  }\n}\n```\n\n### Workflow 2: Multi-Language Evaluation (MultiPL-E)\n\nEvaluate code generation across 18 programming languages.\n\n**Checklist**:\n```\nMulti-Language Evaluation:\n- [ ] Step 1: Generate solutions (host machine)\n- [ ] Step 2: Run evaluation in Docker (safe execution)\n- [ ] Step 3: Compare across languages\n```\n\n**Step 1: Generate solutions on host**\n\n```bash\n# Generate without execution (safe)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --max_length_generation 650 \\\n  --temperature 0.8 \\\n  --n_samples 50 \\\n  --batch_size 50 \\\n  --generation_only \\\n  --save_generations \\\n  --save_generations_path generations_multi.json\n```\n\n**Step 2: Evaluate in Docker container**\n\n```bash\n# Pull the MultiPL-E Docker image\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n\n# Run evaluation inside container\ndocker run -v $(pwd)\u002Fgenerations_multi.json:\u002Fapp\u002Fgenerations.json:ro \\\n  -it evaluation-harness-multiple python3 main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --load_generations_path \u002Fapp\u002Fgenerations.json \\\n  --allow_code_execution \\\n  --n_samples 50\n```\n\n**Supported languages**: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket\n\n### Workflow 3: Instruction-Tuned Model Evaluation\n\nEvaluate chat\u002Finstruction models with proper formatting.\n\n**Checklist**:\n```\nInstruction Model Evaluation:\n- [ ] Step 1: Use instruction-tuned tasks\n- [ ] Step 2: Configure instruction tokens\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Choose instruction tasks**\n\n- **instruct-humaneval**: HumanEval with instruction prompts\n- **humanevalsynthesize-{lang}**: HumanEvalPack synthesis tasks\n\n**Step 2: Configure instruction tokens**\n\n```bash\n# For models with chat templates (e.g., CodeLlama-Instruct)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks instruct-humaneval \\\n  --instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\" \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n**Step 3: HumanEvalPack for instruction models**\n\n```bash\n# Test code synthesis across 6 languages\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks humanevalsynthesize-python,humanevalsynthesize-js \\\n  --prompt instruct \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n### Workflow 4: Compare Multiple Models\n\nBenchmark suite for model comparison.\n\n**Step 1: Create evaluation script**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_models.sh\n\nMODELS=(\n  \"bigcode\u002Fstarcoder2-7b\"\n  \"codellama\u002FCodeLlama-7b-hf\"\n  \"deepseek-ai\u002Fdeepseek-coder-6.7b-base\"\n)\nTASKS=\"humaneval,mbpp\"\n\nfor model in \"${MODELS[@]}\"; do\n  model_name=$(echo $model | tr '\u002F' '-')\n  echo \"Evaluating $model\"\n\n  accelerate launch main.py \\\n    --model $model \\\n    --tasks $TASKS \\\n    --temperature 0.2 \\\n    --n_samples 20 \\\n    --batch_size 20 \\\n    --allow_code_execution \\\n    --metric_output_path results\u002F${model_name}.json\ndone\n```\n\n**Step 2: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\"bigcode-starcoder2-7b\", \"codellama-CodeLlama-7b-hf\", \"deepseek-ai-deepseek-coder-6.7b-base\"]\nresults = []\n\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        results.append({\n            \"Model\": model,\n            \"HumanEval pass@1\": f\"{data['humaneval']['pass@1']:.3f}\",\n            \"MBPP pass@1\": f\"{data['mbpp']['pass@1']:.3f}\"\n        })\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\n## When to Use vs Alternatives\n\n**Use BigCode Evaluation Harness when:**\n- Evaluating **code generation** models specifically\n- Need **multi-language** evaluation (18 languages via MultiPL-E)\n- Testing **functional correctness** with unit tests (pass@k)\n- Benchmarking for **BigCode\u002FHuggingFace leaderboards**\n- Evaluating **fill-in-the-middle** (FIM) capabilities\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)\n- **EvalPlus**: Stricter HumanEval+\u002FMBPP+ with more test cases\n- **SWE-bench**: Real-world GitHub issue resolution\n- **LiveCodeBench**: Contamination-free, continuously updated problems\n- **CodeXGLUE**: Code understanding tasks (clone detection, defect prediction)\n\n## Supported Benchmarks\n\n| Benchmark | Problems | Languages | Metric | Use Case |\n|-----------|----------|-----------|--------|----------|\n| HumanEval | 164 | Python | pass@k | Standard code completion |\n| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |\n| MBPP | 500 | Python | pass@k | Entry-level problems |\n| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |\n| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |\n| APPS | 10,000 | Python | pass@k | Competition-level |\n| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |\n| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis\u002Ffix\u002Fexplain |\n| Mercury | 1,889 | Python | Efficiency | Computational efficiency |\n\n## Common Issues\n\n**Issue: Different results than reported in papers**\n\nCheck these factors:\n```bash\n# 1. Verify n_samples (need 200 for accurate pass@k)\n--n_samples 200\n\n# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)\n--temperature 0.8\n\n# 3. Verify task name matches exactly\n--tasks humaneval  # Not \"human_eval\" or \"HumanEval\"\n\n# 4. Check max_length_generation\n--max_length_generation 512  # Increase for longer problems\n```\n\n**Issue: CUDA out of memory**\n\n```bash\n# Use quantization\n--load_in_8bit\n# OR\n--load_in_4bit\n\n# Reduce batch size\n--batch_size 1\n\n# Set memory limit\n--max_memory_per_gpu \"20GiB\"\n```\n\n**Issue: Code execution hangs or times out**\n\nUse Docker for safe execution:\n```bash\n# Generate on host (no execution)\n--generation_only --save_generations\n\n# Evaluate in Docker\ndocker run ... --allow_code_execution --load_generations_path ...\n```\n\n**Issue: Low scores on instruction models**\n\nEnsure proper instruction formatting:\n```bash\n# Use instruction-specific tasks\n--tasks instruct-humaneval\n\n# Set instruction tokens for your model\n--instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\"\n```\n\n**Issue: MultiPL-E language failures**\n\nUse the dedicated Docker image:\n```bash\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n```\n\n## Command Reference\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--model` | - | HuggingFace model ID or local path |\n| `--tasks` | - | Comma-separated task names |\n| `--n_samples` | 1 | Samples per problem (200 for pass@k) |\n| `--temperature` | 0.2 | Sampling temperature |\n| `--max_length_generation` | 512 | Max tokens (prompt + generation) |\n| `--batch_size` | 1 | Batch size per GPU |\n| `--allow_code_execution` | False | Enable code execution (required) |\n| `--generation_only` | False | Generate without evaluation |\n| `--load_generations_path` | - | Load pre-generated solutions |\n| `--save_generations` | False | Save generated code |\n| `--metric_output_path` | results.json | Output file for metrics |\n| `--load_in_8bit` | False | 8-bit quantization |\n| `--load_in_4bit` | False | 4-bit quantization |\n| `--trust_remote_code` | False | Allow custom model code |\n| `--precision` | fp32 | Model precision (fp32\u002Ffp16\u002Fbf16) |\n\n## Hardware Requirements\n\n| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |\n|------------|-------------|--------------|-------------------------|\n| 7B | 14GB | 6GB | ~30 min (A100) |\n| 13B | 26GB | 10GB | ~1 hour (A100) |\n| 34B | 68GB | 20GB | ~2 hours (A100) |\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\n- **Documentation**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- **BigCode Leaderboard**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbigcode\u002Fbigcode-models-leaderboard\n- **HumanEval Dataset**: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fopenai_humaneval\n- **MultiPL-E**: https:\u002F\u002Fgithub.com\u002Fnuprl\u002FMultiPL-E\n",{"name":5,"tags":24,"author":34,"license":15,"version":35,"description":7,"dependencies":36},[25,26,27,28,29,30,31,32,33],"Evaluation","Code Generation","HumanEval","MBPP","MultiPL-E","Pass@k","BigCode","Benchmarking","Code Models","Orchestra Research","1.0.0",[37,38,39,40],"bigcode-evaluation-harness","transformers>=4.25.1","accelerate>=0.13.2","datasets>=2.6.1","add-x-twitter-scraper-skill","7bc347b2f7dc2dde2761a965585925e0d0171748","cli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fevaluation-bigcode-evaluation-harness",0,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template","https:\u002F\u002Fgithub.com\u002FCarchaloneo22","Ingeniero Informático | Ingeniero en Desarrollo de Software | Consultor TIC en Gestión Publica | Docente Análisis y Desarrollo de Software. ","2026-09-12 11:10:56",{"source":8,"skillName":5,"commands":50,"prompt":60,"zipAvailable":61},[51,54,57],{"tool":52,"command":53},"npx","npx skills add Carchaloneo22\u002FClaude-Code-template --skill evaluating-code-models",{"tool":55,"command":56},"bunx","bunx skills add Carchaloneo22\u002FClaude-Code-template --skill evaluating-code-models",{"tool":58,"command":59},"pnpm","pnpm dlx skills add Carchaloneo22\u002FClaude-Code-template --skill evaluating-code-models","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fevaluation-bigcode-evaluation-harness\u002FSKILL.md",true,[63,69,73],{"path":64,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fbenchmarks.md","REFERENCE","参考文档",9168,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fevaluation-bigcode-evaluation-harness\u002Freferences\u002Fbenchmarks.md",{"path":70,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Fcustom-tasks.md",10378,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fevaluation-bigcode-evaluation-harness\u002Freferences\u002Fcustom-tasks.md",{"path":74,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fissues.md",8085,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fevaluation-bigcode-evaluation-harness\u002Freferences\u002Fissues.md",[78,85,94,101,108,115],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M7CPQFVGKX7H3Q5Y9CN0XAZN","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fzapier-make-patterns\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":89,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7CPQDR2HMKAYHAAY99EA4B2","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","Apache-2.0","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M7CPQ7K5MAQSXEB027HZFM2G","workflow-automation","Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off.  This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation.  Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fworkflow-automation\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M7CPQ34F18JHMEM8NHQS1N7K","trigger-dev","Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Ftrigger-dev\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M7CPPZSXXDB1SAWT5N63ACZT","planning-with-files","Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fplanning-with-files\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":19},"01M7CPPTQSBVWSK0R10FJNMP0F","n8n-workflow-patterns","Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API integration, database operations, AI agent workflows, or scheduled tasks.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fn8n\u002Fn8n-workflow-patterns\u002FSKILL.md",[],[]]