[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7D30PPAXSYJKM46G2SN7ES5":3,"skill-same:01M7D30PPAXSYJKM46G2SN7ES5":74},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":13,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":45,"install":46,"resources":59},"01M7D30PPAXSYJKM46G2SN7ES5","evaluating-code-models",null,"Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.","beyondai\u002Fsandbox","beyondai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F2845958?v=4","01M7D24SBNM47BJGJ9J2HB13BP","01M7D24SCDC9DYE005XH6R553S",0,"MIT","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.cursor\u002Fskills\u002Fbigcode-evaluation-harness\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: evaluating-code-models\ndescription: Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Evaluation, Code Generation, HumanEval, MBPP, MultiPL-E, Pass@k, BigCode, Benchmarking, Code Models]\ndependencies: [bigcode-evaluation-harness, transformers>=4.25.1, accelerate>=0.13.2, datasets>=2.6.1]\n---\n\n# BigCode Evaluation Harness - Code Model Benchmarking\n\n## Quick Start\n\nBigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).\n\n**Installation**:\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness.git\ncd bigcode-evaluation-harness\npip install -e .\naccelerate config\n```\n\n**Evaluate on HumanEval**:\n```bash\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --n_samples 20 \\\n  --batch_size 10 \\\n  --allow_code_execution \\\n  --save_generations\n```\n\n**View available tasks**:\n```bash\npython -c \"from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)\"\n```\n\n## Common Workflows\n\n### Workflow 1: Standard Code Benchmark Evaluation\n\nEvaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).\n\n**Checklist**:\n```\nCode Benchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model and generation\n- [ ] Step 3: Run evaluation with code execution\n- [ ] Step 4: Analyze pass@k results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Python code generation** (most common):\n- **HumanEval**: 164 handwritten problems, function completion\n- **HumanEval+**: Same 164 problems with 80× more tests (stricter)\n- **MBPP**: 500 crowd-sourced problems, entry-level difficulty\n- **MBPP+**: 399 curated problems with 35× more tests\n\n**Multi-language** (18 languages):\n- **MultiPL-E**: HumanEval\u002FMBPP translated to C++, Java, JavaScript, Go, Rust, etc.\n\n**Advanced**:\n- **APPS**: 10,000 problems (introductory\u002Finterview\u002Fcompetition)\n- **DS-1000**: 1,000 data science problems across 7 libraries\n\n**Step 2: Configure model and generation**\n\n```bash\n# Standard HuggingFace model\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --do_sample True \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution\n\n# Quantized model (4-bit)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-34b-hf \\\n  --tasks humaneval \\\n  --load_in_4bit \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n\n# Custom\u002Fprivate model\naccelerate launch main.py \\\n  --model \u002Fpath\u002Fto\u002Fmy-code-model \\\n  --tasks humaneval \\\n  --trust_remote_code \\\n  --use_auth_token \\\n  --allow_code_execution\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full evaluation with pass@k estimation (k=1,10,100)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --temperature 0.8 \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution \\\n  --save_generations \\\n  --metric_output_path results\u002Fstarcoder2-humaneval.json\n```\n\n**Step 4: Analyze results**\n\nResults in `results\u002Fstarcoder2-humaneval.json`:\n```json\n{\n  \"humaneval\": {\n    \"pass@1\": 0.354,\n    \"pass@10\": 0.521,\n    \"pass@100\": 0.689\n  },\n  \"config\": {\n    \"model\": \"bigcode\u002Fstarcoder2-7b\",\n    \"temperature\": 0.8,\n    \"n_samples\": 200\n  }\n}\n```\n\n### Workflow 2: Multi-Language Evaluation (MultiPL-E)\n\nEvaluate code generation across 18 programming languages.\n\n**Checklist**:\n```\nMulti-Language Evaluation:\n- [ ] Step 1: Generate solutions (host machine)\n- [ ] Step 2: Run evaluation in Docker (safe execution)\n- [ ] Step 3: Compare across languages\n```\n\n**Step 1: Generate solutions on host**\n\n```bash\n# Generate without execution (safe)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --max_length_generation 650 \\\n  --temperature 0.8 \\\n  --n_samples 50 \\\n  --batch_size 50 \\\n  --generation_only \\\n  --save_generations \\\n  --save_generations_path generations_multi.json\n```\n\n**Step 2: Evaluate in Docker container**\n\n```bash\n# Pull the MultiPL-E Docker image\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n\n# Run evaluation inside container\ndocker run -v $(pwd)\u002Fgenerations_multi.json:\u002Fapp\u002Fgenerations.json:ro \\\n  -it evaluation-harness-multiple python3 main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --load_generations_path \u002Fapp\u002Fgenerations.json \\\n  --allow_code_execution \\\n  --n_samples 50\n```\n\n**Supported languages**: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket\n\n### Workflow 3: Instruction-Tuned Model Evaluation\n\nEvaluate chat\u002Finstruction models with proper formatting.\n\n**Checklist**:\n```\nInstruction Model Evaluation:\n- [ ] Step 1: Use instruction-tuned tasks\n- [ ] Step 2: Configure instruction tokens\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Choose instruction tasks**\n\n- **instruct-humaneval**: HumanEval with instruction prompts\n- **humanevalsynthesize-{lang}**: HumanEvalPack synthesis tasks\n\n**Step 2: Configure instruction tokens**\n\n```bash\n# For models with chat templates (e.g., CodeLlama-Instruct)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks instruct-humaneval \\\n  --instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\" \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n**Step 3: HumanEvalPack for instruction models**\n\n```bash\n# Test code synthesis across 6 languages\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks humanevalsynthesize-python,humanevalsynthesize-js \\\n  --prompt instruct \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n### Workflow 4: Compare Multiple Models\n\nBenchmark suite for model comparison.\n\n**Step 1: Create evaluation script**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_models.sh\n\nMODELS=(\n  \"bigcode\u002Fstarcoder2-7b\"\n  \"codellama\u002FCodeLlama-7b-hf\"\n  \"deepseek-ai\u002Fdeepseek-coder-6.7b-base\"\n)\nTASKS=\"humaneval,mbpp\"\n\nfor model in \"${MODELS[@]}\"; do\n  model_name=$(echo $model | tr '\u002F' '-')\n  echo \"Evaluating $model\"\n\n  accelerate launch main.py \\\n    --model $model \\\n    --tasks $TASKS \\\n    --temperature 0.2 \\\n    --n_samples 20 \\\n    --batch_size 20 \\\n    --allow_code_execution \\\n    --metric_output_path results\u002F${model_name}.json\ndone\n```\n\n**Step 2: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\"bigcode-starcoder2-7b\", \"codellama-CodeLlama-7b-hf\", \"deepseek-ai-deepseek-coder-6.7b-base\"]\nresults = []\n\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        results.append({\n            \"Model\": model,\n            \"HumanEval pass@1\": f\"{data['humaneval']['pass@1']:.3f}\",\n            \"MBPP pass@1\": f\"{data['mbpp']['pass@1']:.3f}\"\n        })\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\n## When to Use vs Alternatives\n\n**Use BigCode Evaluation Harness when:**\n- Evaluating **code generation** models specifically\n- Need **multi-language** evaluation (18 languages via MultiPL-E)\n- Testing **functional correctness** with unit tests (pass@k)\n- Benchmarking for **BigCode\u002FHuggingFace leaderboards**\n- Evaluating **fill-in-the-middle** (FIM) capabilities\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)\n- **EvalPlus**: Stricter HumanEval+\u002FMBPP+ with more test cases\n- **SWE-bench**: Real-world GitHub issue resolution\n- **LiveCodeBench**: Contamination-free, continuously updated problems\n- **CodeXGLUE**: Code understanding tasks (clone detection, defect prediction)\n\n## Supported Benchmarks\n\n| Benchmark | Problems | Languages | Metric | Use Case |\n|-----------|----------|-----------|--------|----------|\n| HumanEval | 164 | Python | pass@k | Standard code completion |\n| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |\n| MBPP | 500 | Python | pass@k | Entry-level problems |\n| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |\n| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |\n| APPS | 10,000 | Python | pass@k | Competition-level |\n| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |\n| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis\u002Ffix\u002Fexplain |\n| Mercury | 1,889 | Python | Efficiency | Computational efficiency |\n\n## Common Issues\n\n**Issue: Different results than reported in papers**\n\nCheck these factors:\n```bash\n# 1. Verify n_samples (need 200 for accurate pass@k)\n--n_samples 200\n\n# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)\n--temperature 0.8\n\n# 3. Verify task name matches exactly\n--tasks humaneval  # Not \"human_eval\" or \"HumanEval\"\n\n# 4. Check max_length_generation\n--max_length_generation 512  # Increase for longer problems\n```\n\n**Issue: CUDA out of memory**\n\n```bash\n# Use quantization\n--load_in_8bit\n# OR\n--load_in_4bit\n\n# Reduce batch size\n--batch_size 1\n\n# Set memory limit\n--max_memory_per_gpu \"20GiB\"\n```\n\n**Issue: Code execution hangs or times out**\n\nUse Docker for safe execution:\n```bash\n# Generate on host (no execution)\n--generation_only --save_generations\n\n# Evaluate in Docker\ndocker run ... --allow_code_execution --load_generations_path ...\n```\n\n**Issue: Low scores on instruction models**\n\nEnsure proper instruction formatting:\n```bash\n# Use instruction-specific tasks\n--tasks instruct-humaneval\n\n# Set instruction tokens for your model\n--instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\"\n```\n\n**Issue: MultiPL-E language failures**\n\nUse the dedicated Docker image:\n```bash\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n```\n\n## Command Reference\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--model` | - | HuggingFace model ID or local path |\n| `--tasks` | - | Comma-separated task names |\n| `--n_samples` | 1 | Samples per problem (200 for pass@k) |\n| `--temperature` | 0.2 | Sampling temperature |\n| `--max_length_generation` | 512 | Max tokens (prompt + generation) |\n| `--batch_size` | 1 | Batch size per GPU |\n| `--allow_code_execution` | False | Enable code execution (required) |\n| `--generation_only` | False | Generate without evaluation |\n| `--load_generations_path` | - | Load pre-generated solutions |\n| `--save_generations` | False | Save generated code |\n| `--metric_output_path` | results.json | Output file for metrics |\n| `--load_in_8bit` | False | 8-bit quantization |\n| `--load_in_4bit` | False | 4-bit quantization |\n| `--trust_remote_code` | False | Allow custom model code |\n| `--precision` | fp32 | Model precision (fp32\u002Ffp16\u002Fbf16) |\n\n## Hardware Requirements\n\n| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |\n|------------|-------------|--------------|-------------------------|\n| 7B | 14GB | 6GB | ~30 min (A100) |\n| 13B | 26GB | 10GB | ~1 hour (A100) |\n| 34B | 68GB | 20GB | ~2 hours (A100) |\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\n- **Documentation**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- **BigCode Leaderboard**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbigcode\u002Fbigcode-models-leaderboard\n- **HumanEval Dataset**: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fopenai_humaneval\n- **MultiPL-E**: https:\u002F\u002Fgithub.com\u002Fnuprl\u002FMultiPL-E\n","# BigCode Evaluation Harness - Code Model Benchmarking\n\n## Quick Start\n\nBigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).\n\n**Installation**:\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness.git\ncd bigcode-evaluation-harness\npip install -e .\naccelerate config\n```\n\n**Evaluate on HumanEval**:\n```bash\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --n_samples 20 \\\n  --batch_size 10 \\\n  --allow_code_execution \\\n  --save_generations\n```\n\n**View available tasks**:\n```bash\npython -c \"from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)\"\n```\n\n## Common Workflows\n\n### Workflow 1: Standard Code Benchmark Evaluation\n\nEvaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).\n\n**Checklist**:\n```\nCode Benchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model and generation\n- [ ] Step 3: Run evaluation with code execution\n- [ ] Step 4: Analyze pass@k results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Python code generation** (most common):\n- **HumanEval**: 164 handwritten problems, function completion\n- **HumanEval+**: Same 164 problems with 80× more tests (stricter)\n- **MBPP**: 500 crowd-sourced problems, entry-level difficulty\n- **MBPP+**: 399 curated problems with 35× more tests\n\n**Multi-language** (18 languages):\n- **MultiPL-E**: HumanEval\u002FMBPP translated to C++, Java, JavaScript, Go, Rust, etc.\n\n**Advanced**:\n- **APPS**: 10,000 problems (introductory\u002Finterview\u002Fcompetition)\n- **DS-1000**: 1,000 data science problems across 7 libraries\n\n**Step 2: Configure model and generation**\n\n```bash\n# Standard HuggingFace model\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --do_sample True \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution\n\n# Quantized model (4-bit)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-34b-hf \\\n  --tasks humaneval \\\n  --load_in_4bit \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n\n# Custom\u002Fprivate model\naccelerate launch main.py \\\n  --model \u002Fpath\u002Fto\u002Fmy-code-model \\\n  --tasks humaneval \\\n  --trust_remote_code \\\n  --use_auth_token \\\n  --allow_code_execution\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full evaluation with pass@k estimation (k=1,10,100)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --temperature 0.8 \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution \\\n  --save_generations \\\n  --metric_output_path results\u002Fstarcoder2-humaneval.json\n```\n\n**Step 4: Analyze results**\n\nResults in `results\u002Fstarcoder2-humaneval.json`:\n```json\n{\n  \"humaneval\": {\n    \"pass@1\": 0.354,\n    \"pass@10\": 0.521,\n    \"pass@100\": 0.689\n  },\n  \"config\": {\n    \"model\": \"bigcode\u002Fstarcoder2-7b\",\n    \"temperature\": 0.8,\n    \"n_samples\": 200\n  }\n}\n```\n\n### Workflow 2: Multi-Language Evaluation (MultiPL-E)\n\nEvaluate code generation across 18 programming languages.\n\n**Checklist**:\n```\nMulti-Language Evaluation:\n- [ ] Step 1: Generate solutions (host machine)\n- [ ] Step 2: Run evaluation in Docker (safe execution)\n- [ ] Step 3: Compare across languages\n```\n\n**Step 1: Generate solutions on host**\n\n```bash\n# Generate without execution (safe)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --max_length_generation 650 \\\n  --temperature 0.8 \\\n  --n_samples 50 \\\n  --batch_size 50 \\\n  --generation_only \\\n  --save_generations \\\n  --save_generations_path generations_multi.json\n```\n\n**Step 2: Evaluate in Docker container**\n\n```bash\n# Pull the MultiPL-E Docker image\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n\n# Run evaluation inside container\ndocker run -v $(pwd)\u002Fgenerations_multi.json:\u002Fapp\u002Fgenerations.json:ro \\\n  -it evaluation-harness-multiple python3 main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --load_generations_path \u002Fapp\u002Fgenerations.json \\\n  --allow_code_execution \\\n  --n_samples 50\n```\n\n**Supported languages**: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket\n\n### Workflow 3: Instruction-Tuned Model Evaluation\n\nEvaluate chat\u002Finstruction models with proper formatting.\n\n**Checklist**:\n```\nInstruction Model Evaluation:\n- [ ] Step 1: Use instruction-tuned tasks\n- [ ] Step 2: Configure instruction tokens\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Choose instruction tasks**\n\n- **instruct-humaneval**: HumanEval with instruction prompts\n- **humanevalsynthesize-{lang}**: HumanEvalPack synthesis tasks\n\n**Step 2: Configure instruction tokens**\n\n```bash\n# For models with chat templates (e.g., CodeLlama-Instruct)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks instruct-humaneval \\\n  --instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\" \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n**Step 3: HumanEvalPack for instruction models**\n\n```bash\n# Test code synthesis across 6 languages\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks humanevalsynthesize-python,humanevalsynthesize-js \\\n  --prompt instruct \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n### Workflow 4: Compare Multiple Models\n\nBenchmark suite for model comparison.\n\n**Step 1: Create evaluation script**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_models.sh\n\nMODELS=(\n  \"bigcode\u002Fstarcoder2-7b\"\n  \"codellama\u002FCodeLlama-7b-hf\"\n  \"deepseek-ai\u002Fdeepseek-coder-6.7b-base\"\n)\nTASKS=\"humaneval,mbpp\"\n\nfor model in \"${MODELS[@]}\"; do\n  model_name=$(echo $model | tr '\u002F' '-')\n  echo \"Evaluating $model\"\n\n  accelerate launch main.py \\\n    --model $model \\\n    --tasks $TASKS \\\n    --temperature 0.2 \\\n    --n_samples 20 \\\n    --batch_size 20 \\\n    --allow_code_execution \\\n    --metric_output_path results\u002F${model_name}.json\ndone\n```\n\n**Step 2: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\"bigcode-starcoder2-7b\", \"codellama-CodeLlama-7b-hf\", \"deepseek-ai-deepseek-coder-6.7b-base\"]\nresults = []\n\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        results.append({\n            \"Model\": model,\n            \"HumanEval pass@1\": f\"{data['humaneval']['pass@1']:.3f}\",\n            \"MBPP pass@1\": f\"{data['mbpp']['pass@1']:.3f}\"\n        })\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\n## When to Use vs Alternatives\n\n**Use BigCode Evaluation Harness when:**\n- Evaluating **code generation** models specifically\n- Need **multi-language** evaluation (18 languages via MultiPL-E)\n- Testing **functional correctness** with unit tests (pass@k)\n- Benchmarking for **BigCode\u002FHuggingFace leaderboards**\n- Evaluating **fill-in-the-middle** (FIM) capabilities\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)\n- **EvalPlus**: Stricter HumanEval+\u002FMBPP+ with more test cases\n- **SWE-bench**: Real-world GitHub issue resolution\n- **LiveCodeBench**: Contamination-free, continuously updated problems\n- **CodeXGLUE**: Code understanding tasks (clone detection, defect prediction)\n\n## Supported Benchmarks\n\n| Benchmark | Problems | Languages | Metric | Use Case |\n|-----------|----------|-----------|--------|----------|\n| HumanEval | 164 | Python | pass@k | Standard code completion |\n| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |\n| MBPP | 500 | Python | pass@k | Entry-level problems |\n| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |\n| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |\n| APPS | 10,000 | Python | pass@k | Competition-level |\n| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |\n| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis\u002Ffix\u002Fexplain |\n| Mercury | 1,889 | Python | Efficiency | Computational efficiency |\n\n## Common Issues\n\n**Issue: Different results than reported in papers**\n\nCheck these factors:\n```bash\n# 1. Verify n_samples (need 200 for accurate pass@k)\n--n_samples 200\n\n# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)\n--temperature 0.8\n\n# 3. Verify task name matches exactly\n--tasks humaneval  # Not \"human_eval\" or \"HumanEval\"\n\n# 4. Check max_length_generation\n--max_length_generation 512  # Increase for longer problems\n```\n\n**Issue: CUDA out of memory**\n\n```bash\n# Use quantization\n--load_in_8bit\n# OR\n--load_in_4bit\n\n# Reduce batch size\n--batch_size 1\n\n# Set memory limit\n--max_memory_per_gpu \"20GiB\"\n```\n\n**Issue: Code execution hangs or times out**\n\nUse Docker for safe execution:\n```bash\n# Generate on host (no execution)\n--generation_only --save_generations\n\n# Evaluate in Docker\ndocker run ... --allow_code_execution --load_generations_path ...\n```\n\n**Issue: Low scores on instruction models**\n\nEnsure proper instruction formatting:\n```bash\n# Use instruction-specific tasks\n--tasks instruct-humaneval\n\n# Set instruction tokens for your model\n--instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\"\n```\n\n**Issue: MultiPL-E language failures**\n\nUse the dedicated Docker image:\n```bash\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n```\n\n## Command Reference\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--model` | - | HuggingFace model ID or local path |\n| `--tasks` | - | Comma-separated task names |\n| `--n_samples` | 1 | Samples per problem (200 for pass@k) |\n| `--temperature` | 0.2 | Sampling temperature |\n| `--max_length_generation` | 512 | Max tokens (prompt + generation) |\n| `--batch_size` | 1 | Batch size per GPU |\n| `--allow_code_execution` | False | Enable code execution (required) |\n| `--generation_only` | False | Generate without evaluation |\n| `--load_generations_path` | - | Load pre-generated solutions |\n| `--save_generations` | False | Save generated code |\n| `--metric_output_path` | results.json | Output file for metrics |\n| `--load_in_8bit` | False | 8-bit quantization |\n| `--load_in_4bit` | False | 4-bit quantization |\n| `--trust_remote_code` | False | Allow custom model code |\n| `--precision` | fp32 | Model precision (fp32\u002Ffp16\u002Fbf16) |\n\n## Hardware Requirements\n\n| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |\n|------------|-------------|--------------|-------------------------|\n| 7B | 14GB | 6GB | ~30 min (A100) |\n| 13B | 26GB | 10GB | ~1 hour (A100) |\n| 34B | 68GB | 20GB | ~2 hours (A100) |\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\n- **Documentation**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- **BigCode Leaderboard**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbigcode\u002Fbigcode-models-leaderboard\n- **HumanEval Dataset**: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fopenai_humaneval\n- **MultiPL-E**: https:\u002F\u002Fgithub.com\u002Fnuprl\u002FMultiPL-E\n",{"name":5,"tags":23,"author":33,"license":14,"version":34,"description":7,"dependencies":35},[24,25,26,27,28,29,30,31,32],"Evaluation","Code Generation","HumanEval","MBPP","MultiPL-E","Pass@k","BigCode","Benchmarking","Code Models","Orchestra Research","1.0.0",[36,37,38,39],"bigcode-evaluation-harness","transformers>=4.25.1","accelerate>=0.13.2","datasets>=2.6.1","main","46e03791269608c5ac886a64c08f5d94147e8786","labs\u002F.cursor\u002Fskills\u002Fbigcode-evaluation-harness","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox","https:\u002F\u002Fgithub.com\u002Fbeyondai","2026-11-08 04:18:42",{"source":8,"skillName":5,"commands":47,"prompt":57,"zipAvailable":58},[48,51,54],{"tool":49,"command":50},"npx","npx skills add beyondai\u002Fsandbox --skill evaluating-code-models",{"tool":52,"command":53},"bunx","bunx skills add beyondai\u002Fsandbox --skill evaluating-code-models",{"tool":55,"command":56},"pnpm","pnpm dlx skills add beyondai\u002Fsandbox --skill evaluating-code-models","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.cursor\u002Fskills\u002Fbigcode-evaluation-harness\u002FSKILL.md",true,[60,66,70],{"path":61,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":64,"sourceHtmlUrl":65},"references\u002Fbenchmarks.md","REFERENCE","参考文档",9168,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.cursor\u002Fskills\u002Fbigcode-evaluation-harness\u002Freferences\u002Fbenchmarks.md",{"path":67,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fcustom-tasks.md",10378,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.cursor\u002Fskills\u002Fbigcode-evaluation-harness\u002Freferences\u002Fcustom-tasks.md",{"path":71,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fissues.md",8085,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.cursor\u002Fskills\u002Fbigcode-evaluation-harness\u002Freferences\u002Fissues.md",[75,82,89,96,103,110],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":18},"01M7D3GMPGJZRG8EPY404YF8TR","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":18},"01M7D3GFZH440V2J532BSZMPVP","serving-llms-vllm","Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Fvllm\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":18},"01M7D3GAE8MQCPGF72EP9G9GKY","verl-rl-training","Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Fverl\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":18},"01M7D3G7KKNE5RGNYJNXV4F0MM","unsloth","Expert guidance for fast fine-tuning with Unsloth - 2-5x faster training, 50-80% less memory, LoRA\u002FQLoRA optimization","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Funsloth\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":18},"01M7D3G2RCZQ08GS0KAW5CBWSC","fine-tuning-with-trl","Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Ftrl-fine-tuning\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":18},"01M7D3FX08BPB172CYGM597NZF","torchforge-rl-training","Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Ftorchforge\u002FSKILL.md",[],[]]