[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XG91YRRF8WKB819FDZ3TVP":3,"skill-same:01M6XG91YRRF8WKB819FDZ3TVP":95},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":44,"linkOnly":45,"markdownContent":46,"markdownBody":47,"frontMatter":48,"refName":61,"commitSha":62,"skillDirectory":63,"forksCount":14,"repositoryHtmlUrl":64,"repositoryHomepageUrl":6,"ownerHtmlUrl":65,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":66,"install":67,"resources":80},"01M6XG91YRRF8WKB819FDZ3TVP","evaluating-code-models",null,"Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.","fadrienne\u002FMyGitHub","fadrienne","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F197874571?v=4","01M6X64V8VWCRSMFZ4EPS2BWXG","Francesca Adrienne","01M6X64VD5MW4F9PHDWQKS7ZAF",0,"MIT","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fbigcode-evaluation-harness\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],"evaluating code models","code","models","Code Models","when","testing","multipl","Accelerate","synthetic","project","across","Transformers","MultiPL-E","MBPP","HumanEval","Code Generation","Evaluation","training","benchmarking","support","2026-11-10 10:12:30",false,"---\nname: evaluating-code-models\ndescription: Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.\ncategory: ml-training\nversion: 1.0.0\nauthor: Synthetic Sciences\nlicense: MIT\ntags: [Evaluation, Code Generation, HumanEval, MBPP, MultiPL-E, Pass@k, BigCode, Benchmarking, Code Models]\ndependencies: [bigcode-evaluation-harness, transformers>=4.25.1, accelerate>=0.13.2, datasets>=2.6.1]\n---\n\n# BigCode Evaluation Harness - Code Model Benchmarking\n\n## Quick Start\n\nBigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).\n\n**Installation**:\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness.git\ncd bigcode-evaluation-harness\npip install -e .\naccelerate config\n```\n\n**Evaluate on HumanEval**:\n```bash\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --n_samples 20 \\\n  --batch_size 10 \\\n  --allow_code_execution \\\n  --save_generations\n```\n\n**View available tasks**:\n```bash\npython -c \"from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)\"\n```\n\n## Common Workflows\n\n### Workflow 1: Standard Code Benchmark Evaluation\n\nEvaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).\n\n**Checklist**:\n```\nCode Benchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model and generation\n- [ ] Step 3: Run evaluation with code execution\n- [ ] Step 4: Analyze pass@k results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Python code generation** (most common):\n- **HumanEval**: 164 handwritten problems, function completion\n- **HumanEval+**: Same 164 problems with 80× more tests (stricter)\n- **MBPP**: 500 crowd-sourced problems, entry-level difficulty\n- **MBPP+**: 399 curated problems with 35× more tests\n\n**Multi-language** (18 languages):\n- **MultiPL-E**: HumanEval\u002FMBPP translated to C++, Java, JavaScript, Go, Rust, etc.\n\n**Advanced**:\n- **APPS**: 10,000 problems (introductory\u002Finterview\u002Fcompetition)\n- **DS-1000**: 1,000 data science problems across 7 libraries\n\n**Step 2: Configure model and generation**\n\n```bash\n# Standard HuggingFace model\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --do_sample True \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution\n\n# Quantized model (4-bit)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-34b-hf \\\n  --tasks humaneval \\\n  --load_in_4bit \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n\n# Custom\u002Fprivate model\naccelerate launch main.py \\\n  --model \u002Fpath\u002Fto\u002Fmy-code-model \\\n  --tasks humaneval \\\n  --trust_remote_code \\\n  --use_auth_token \\\n  --allow_code_execution\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full evaluation with pass@k estimation (k=1,10,100)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --temperature 0.8 \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution \\\n  --save_generations \\\n  --metric_output_path results\u002Fstarcoder2-humaneval.json\n```\n\n**Step 4: Analyze results**\n\nResults in `results\u002Fstarcoder2-humaneval.json`:\n```json\n{\n  \"humaneval\": {\n    \"pass@1\": 0.354,\n    \"pass@10\": 0.521,\n    \"pass@100\": 0.689\n  },\n  \"config\": {\n    \"model\": \"bigcode\u002Fstarcoder2-7b\",\n    \"temperature\": 0.8,\n    \"n_samples\": 200\n  }\n}\n```\n\n### Workflow 2: Multi-Language Evaluation (MultiPL-E)\n\nEvaluate code generation across 18 programming languages.\n\n**Checklist**:\n```\nMulti-Language Evaluation:\n- [ ] Step 1: Generate solutions (host machine)\n- [ ] Step 2: Run evaluation in Docker (safe execution)\n- [ ] Step 3: Compare across languages\n```\n\n**Step 1: Generate solutions on host**\n\n```bash\n# Generate without execution (safe)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --max_length_generation 650 \\\n  --temperature 0.8 \\\n  --n_samples 50 \\\n  --batch_size 50 \\\n  --generation_only \\\n  --save_generations \\\n  --save_generations_path generations_multi.json\n```\n\n**Step 2: Evaluate in Docker container**\n\n```bash\n# Pull the MultiPL-E Docker image\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n\n# Run evaluation inside container\ndocker run -v $(pwd)\u002Fgenerations_multi.json:\u002Fapp\u002Fgenerations.json:ro \\\n  -it evaluation-harness-multiple python3 main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --load_generations_path \u002Fapp\u002Fgenerations.json \\\n  --allow_code_execution \\\n  --n_samples 50\n```\n\n**Supported languages**: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket\n\n### Workflow 3: Instruction-Tuned Model Evaluation\n\nEvaluate chat\u002Finstruction models with proper formatting.\n\n**Checklist**:\n```\nInstruction Model Evaluation:\n- [ ] Step 1: Use instruction-tuned tasks\n- [ ] Step 2: Configure instruction tokens\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Choose instruction tasks**\n\n- **instruct-humaneval**: HumanEval with instruction prompts\n- **humanevalsynthesize-{lang}**: HumanEvalPack synthesis tasks\n\n**Step 2: Configure instruction tokens**\n\n```bash\n# For models with chat templates (e.g., CodeLlama-Instruct)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks instruct-humaneval \\\n  --instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\" \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n**Step 3: HumanEvalPack for instruction models**\n\n```bash\n# Test code synthesis across 6 languages\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks humanevalsynthesize-python,humanevalsynthesize-js \\\n  --prompt instruct \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n### Workflow 4: Compare Multiple Models\n\nBenchmark suite for model comparison.\n\n**Step 1: Create evaluation script**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_models.sh\n\nMODELS=(\n  \"bigcode\u002Fstarcoder2-7b\"\n  \"codellama\u002FCodeLlama-7b-hf\"\n  \"deepseek-ai\u002Fdeepseek-coder-6.7b-base\"\n)\nTASKS=\"humaneval,mbpp\"\n\nfor model in \"${MODELS[@]}\"; do\n  model_name=$(echo $model | tr '\u002F' '-')\n  echo \"Evaluating $model\"\n\n  accelerate launch main.py \\\n    --model $model \\\n    --tasks $TASKS \\\n    --temperature 0.2 \\\n    --n_samples 20 \\\n    --batch_size 20 \\\n    --allow_code_execution \\\n    --metric_output_path results\u002F${model_name}.json\ndone\n```\n\n**Step 2: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\"bigcode-starcoder2-7b\", \"codellama-CodeLlama-7b-hf\", \"deepseek-ai-deepseek-coder-6.7b-base\"]\nresults = []\n\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        results.append({\n            \"Model\": model,\n            \"HumanEval pass@1\": f\"{data['humaneval']['pass@1']:.3f}\",\n            \"MBPP pass@1\": f\"{data['mbpp']['pass@1']:.3f}\"\n        })\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\n## When to Use vs Alternatives\n\n**Use BigCode Evaluation Harness when:**\n- Evaluating **code generation** models specifically\n- Need **multi-language** evaluation (18 languages via MultiPL-E)\n- Testing **functional correctness** with unit tests (pass@k)\n- Benchmarking for **BigCode\u002FHuggingFace leaderboards**\n- Evaluating **fill-in-the-middle** (FIM) capabilities\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)\n- **EvalPlus**: Stricter HumanEval+\u002FMBPP+ with more test cases\n- **SWE-bench**: Real-world GitHub issue resolution\n- **LiveCodeBench**: Contamination-free, continuously updated problems\n- **CodeXGLUE**: Code understanding tasks (clone detection, defect prediction)\n\n## Supported Benchmarks\n\n| Benchmark | Problems | Languages | Metric | Use Case |\n|-----------|----------|-----------|--------|----------|\n| HumanEval | 164 | Python | pass@k | Standard code completion |\n| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |\n| MBPP | 500 | Python | pass@k | Entry-level problems |\n| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |\n| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |\n| APPS | 10,000 | Python | pass@k | Competition-level |\n| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |\n| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis\u002Ffix\u002Fexplain |\n| Mercury | 1,889 | Python | Efficiency | Computational efficiency |\n\n## Common Issues\n\n**Issue: Different results than reported in papers**\n\nCheck these factors:\n```bash\n# 1. Verify n_samples (need 200 for accurate pass@k)\n--n_samples 200\n\n# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)\n--temperature 0.8\n\n# 3. Verify task name matches exactly\n--tasks humaneval  # Not \"human_eval\" or \"HumanEval\"\n\n# 4. Check max_length_generation\n--max_length_generation 512  # Increase for longer problems\n```\n\n**Issue: CUDA out of memory**\n\n```bash\n# Use quantization\n--load_in_8bit\n# OR\n--load_in_4bit\n\n# Reduce batch size\n--batch_size 1\n\n# Set memory limit\n--max_memory_per_gpu \"20GiB\"\n```\n\n**Issue: Code execution hangs or times out**\n\nUse Docker for safe execution:\n```bash\n# Generate on host (no execution)\n--generation_only --save_generations\n\n# Evaluate in Docker\ndocker run ... --allow_code_execution --load_generations_path ...\n```\n\n**Issue: Low scores on instruction models**\n\nEnsure proper instruction formatting:\n```bash\n# Use instruction-specific tasks\n--tasks instruct-humaneval\n\n# Set instruction tokens for your model\n--instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\"\n```\n\n**Issue: MultiPL-E language failures**\n\nUse the dedicated Docker image:\n```bash\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n```\n\n## Command Reference\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--model` | - | HuggingFace model ID or local path |\n| `--tasks` | - | Comma-separated task names |\n| `--n_samples` | 1 | Samples per problem (200 for pass@k) |\n| `--temperature` | 0.2 | Sampling temperature |\n| `--max_length_generation` | 512 | Max tokens (prompt + generation) |\n| `--batch_size` | 1 | Batch size per GPU |\n| `--allow_code_execution` | False | Enable code execution (required) |\n| `--generation_only` | False | Generate without evaluation |\n| `--load_generations_path` | - | Load pre-generated solutions |\n| `--save_generations` | False | Save generated code |\n| `--metric_output_path` | results.json | Output file for metrics |\n| `--load_in_8bit` | False | 8-bit quantization |\n| `--load_in_4bit` | False | 4-bit quantization |\n| `--trust_remote_code` | False | Allow custom model code |\n| `--precision` | fp32 | Model precision (fp32\u002Ffp16\u002Fbf16) |\n\n## Hardware Requirements\n\n| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |\n|------------|-------------|--------------|-------------------------|\n| 7B | 14GB | 6GB | ~30 min (A100) |\n| 13B | 26GB | 10GB | ~1 hour (A100) |\n| 34B | 68GB | 20GB | ~2 hours (A100) |\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\n- **Documentation**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- **BigCode Leaderboard**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbigcode\u002Fbigcode-models-leaderboard\n- **HumanEval Dataset**: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fopenai_humaneval\n- **MultiPL-E**: https:\u002F\u002Fgithub.com\u002Fnuprl\u002FMultiPL-E\n","# BigCode Evaluation Harness - Code Model Benchmarking\n\n## Quick Start\n\nBigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).\n\n**Installation**:\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness.git\ncd bigcode-evaluation-harness\npip install -e .\naccelerate config\n```\n\n**Evaluate on HumanEval**:\n```bash\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --n_samples 20 \\\n  --batch_size 10 \\\n  --allow_code_execution \\\n  --save_generations\n```\n\n**View available tasks**:\n```bash\npython -c \"from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)\"\n```\n\n## Common Workflows\n\n### Workflow 1: Standard Code Benchmark Evaluation\n\nEvaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).\n\n**Checklist**:\n```\nCode Benchmark Evaluation:\n- [ ] Step 1: Choose benchmark suite\n- [ ] Step 2: Configure model and generation\n- [ ] Step 3: Run evaluation with code execution\n- [ ] Step 4: Analyze pass@k results\n```\n\n**Step 1: Choose benchmark suite**\n\n**Python code generation** (most common):\n- **HumanEval**: 164 handwritten problems, function completion\n- **HumanEval+**: Same 164 problems with 80× more tests (stricter)\n- **MBPP**: 500 crowd-sourced problems, entry-level difficulty\n- **MBPP+**: 399 curated problems with 35× more tests\n\n**Multi-language** (18 languages):\n- **MultiPL-E**: HumanEval\u002FMBPP translated to C++, Java, JavaScript, Go, Rust, etc.\n\n**Advanced**:\n- **APPS**: 10,000 problems (introductory\u002Finterview\u002Fcompetition)\n- **DS-1000**: 1,000 data science problems across 7 libraries\n\n**Step 2: Configure model and generation**\n\n```bash\n# Standard HuggingFace model\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --max_length_generation 512 \\\n  --temperature 0.2 \\\n  --do_sample True \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution\n\n# Quantized model (4-bit)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-34b-hf \\\n  --tasks humaneval \\\n  --load_in_4bit \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n\n# Custom\u002Fprivate model\naccelerate launch main.py \\\n  --model \u002Fpath\u002Fto\u002Fmy-code-model \\\n  --tasks humaneval \\\n  --trust_remote_code \\\n  --use_auth_token \\\n  --allow_code_execution\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Full evaluation with pass@k estimation (k=1,10,100)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks humaneval \\\n  --temperature 0.8 \\\n  --n_samples 200 \\\n  --batch_size 50 \\\n  --allow_code_execution \\\n  --save_generations \\\n  --metric_output_path results\u002Fstarcoder2-humaneval.json\n```\n\n**Step 4: Analyze results**\n\nResults in `results\u002Fstarcoder2-humaneval.json`:\n```json\n{\n  \"humaneval\": {\n    \"pass@1\": 0.354,\n    \"pass@10\": 0.521,\n    \"pass@100\": 0.689\n  },\n  \"config\": {\n    \"model\": \"bigcode\u002Fstarcoder2-7b\",\n    \"temperature\": 0.8,\n    \"n_samples\": 200\n  }\n}\n```\n\n### Workflow 2: Multi-Language Evaluation (MultiPL-E)\n\nEvaluate code generation across 18 programming languages.\n\n**Checklist**:\n```\nMulti-Language Evaluation:\n- [ ] Step 1: Generate solutions (host machine)\n- [ ] Step 2: Run evaluation in Docker (safe execution)\n- [ ] Step 3: Compare across languages\n```\n\n**Step 1: Generate solutions on host**\n\n```bash\n# Generate without execution (safe)\naccelerate launch main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --max_length_generation 650 \\\n  --temperature 0.8 \\\n  --n_samples 50 \\\n  --batch_size 50 \\\n  --generation_only \\\n  --save_generations \\\n  --save_generations_path generations_multi.json\n```\n\n**Step 2: Evaluate in Docker container**\n\n```bash\n# Pull the MultiPL-E Docker image\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n\n# Run evaluation inside container\ndocker run -v $(pwd)\u002Fgenerations_multi.json:\u002Fapp\u002Fgenerations.json:ro \\\n  -it evaluation-harness-multiple python3 main.py \\\n  --model bigcode\u002Fstarcoder2-7b \\\n  --tasks multiple-py,multiple-js,multiple-java,multiple-cpp \\\n  --load_generations_path \u002Fapp\u002Fgenerations.json \\\n  --allow_code_execution \\\n  --n_samples 50\n```\n\n**Supported languages**: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket\n\n### Workflow 3: Instruction-Tuned Model Evaluation\n\nEvaluate chat\u002Finstruction models with proper formatting.\n\n**Checklist**:\n```\nInstruction Model Evaluation:\n- [ ] Step 1: Use instruction-tuned tasks\n- [ ] Step 2: Configure instruction tokens\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Choose instruction tasks**\n\n- **instruct-humaneval**: HumanEval with instruction prompts\n- **humanevalsynthesize-{lang}**: HumanEvalPack synthesis tasks\n\n**Step 2: Configure instruction tokens**\n\n```bash\n# For models with chat templates (e.g., CodeLlama-Instruct)\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks instruct-humaneval \\\n  --instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\" \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n**Step 3: HumanEvalPack for instruction models**\n\n```bash\n# Test code synthesis across 6 languages\naccelerate launch main.py \\\n  --model codellama\u002FCodeLlama-7b-Instruct-hf \\\n  --tasks humanevalsynthesize-python,humanevalsynthesize-js \\\n  --prompt instruct \\\n  --max_length_generation 512 \\\n  --allow_code_execution\n```\n\n### Workflow 4: Compare Multiple Models\n\nBenchmark suite for model comparison.\n\n**Step 1: Create evaluation script**\n\n```bash\n#!\u002Fbin\u002Fbash\n# eval_models.sh\n\nMODELS=(\n  \"bigcode\u002Fstarcoder2-7b\"\n  \"codellama\u002FCodeLlama-7b-hf\"\n  \"deepseek-ai\u002Fdeepseek-coder-6.7b-base\"\n)\nTASKS=\"humaneval,mbpp\"\n\nfor model in \"${MODELS[@]}\"; do\n  model_name=$(echo $model | tr '\u002F' '-')\n  echo \"Evaluating $model\"\n\n  accelerate launch main.py \\\n    --model $model \\\n    --tasks $TASKS \\\n    --temperature 0.2 \\\n    --n_samples 20 \\\n    --batch_size 20 \\\n    --allow_code_execution \\\n    --metric_output_path results\u002F${model_name}.json\ndone\n```\n\n**Step 2: Generate comparison table**\n\n```python\nimport json\nimport pandas as pd\n\nmodels = [\"bigcode-starcoder2-7b\", \"codellama-CodeLlama-7b-hf\", \"deepseek-ai-deepseek-coder-6.7b-base\"]\nresults = []\n\nfor model in models:\n    with open(f\"results\u002F{model}.json\") as f:\n        data = json.load(f)\n        results.append({\n            \"Model\": model,\n            \"HumanEval pass@1\": f\"{data['humaneval']['pass@1']:.3f}\",\n            \"MBPP pass@1\": f\"{data['mbpp']['pass@1']:.3f}\"\n        })\n\ndf = pd.DataFrame(results)\nprint(df.to_markdown(index=False))\n```\n\n## When to Use vs Alternatives\n\n**Use BigCode Evaluation Harness when:**\n- Evaluating **code generation** models specifically\n- Need **multi-language** evaluation (18 languages via MultiPL-E)\n- Testing **functional correctness** with unit tests (pass@k)\n- Benchmarking for **BigCode\u002FHuggingFace leaderboards**\n- Evaluating **fill-in-the-middle** (FIM) capabilities\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)\n- **EvalPlus**: Stricter HumanEval+\u002FMBPP+ with more test cases\n- **SWE-bench**: Real-world GitHub issue resolution\n- **LiveCodeBench**: Contamination-free, continuously updated problems\n- **CodeXGLUE**: Code understanding tasks (clone detection, defect prediction)\n\n## Supported Benchmarks\n\n| Benchmark | Problems | Languages | Metric | Use Case |\n|-----------|----------|-----------|--------|----------|\n| HumanEval | 164 | Python | pass@k | Standard code completion |\n| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |\n| MBPP | 500 | Python | pass@k | Entry-level problems |\n| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |\n| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |\n| APPS | 10,000 | Python | pass@k | Competition-level |\n| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |\n| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis\u002Ffix\u002Fexplain |\n| Mercury | 1,889 | Python | Efficiency | Computational efficiency |\n\n## Common Issues\n\n**Issue: Different results than reported in papers**\n\nCheck these factors:\n```bash\n# 1. Verify n_samples (need 200 for accurate pass@k)\n--n_samples 200\n\n# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)\n--temperature 0.8\n\n# 3. Verify task name matches exactly\n--tasks humaneval  # Not \"human_eval\" or \"HumanEval\"\n\n# 4. Check max_length_generation\n--max_length_generation 512  # Increase for longer problems\n```\n\n**Issue: CUDA out of memory**\n\n```bash\n# Use quantization\n--load_in_8bit\n# OR\n--load_in_4bit\n\n# Reduce batch size\n--batch_size 1\n\n# Set memory limit\n--max_memory_per_gpu \"20GiB\"\n```\n\n**Issue: Code execution hangs or times out**\n\nUse Docker for safe execution:\n```bash\n# Generate on host (no execution)\n--generation_only --save_generations\n\n# Evaluate in Docker\ndocker run ... --allow_code_execution --load_generations_path ...\n```\n\n**Issue: Low scores on instruction models**\n\nEnsure proper instruction formatting:\n```bash\n# Use instruction-specific tasks\n--tasks instruct-humaneval\n\n# Set instruction tokens for your model\n--instruction_tokens \"\u003Cs>[INST],\u003C\u002Fs>,[\u002FINST]\"\n```\n\n**Issue: MultiPL-E language failures**\n\nUse the dedicated Docker image:\n```bash\ndocker pull ghcr.io\u002Fbigcode-project\u002Fevaluation-harness-multiple\n```\n\n## Command Reference\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--model` | - | HuggingFace model ID or local path |\n| `--tasks` | - | Comma-separated task names |\n| `--n_samples` | 1 | Samples per problem (200 for pass@k) |\n| `--temperature` | 0.2 | Sampling temperature |\n| `--max_length_generation` | 512 | Max tokens (prompt + generation) |\n| `--batch_size` | 1 | Batch size per GPU |\n| `--allow_code_execution` | False | Enable code execution (required) |\n| `--generation_only` | False | Generate without evaluation |\n| `--load_generations_path` | - | Load pre-generated solutions |\n| `--save_generations` | False | Save generated code |\n| `--metric_output_path` | results.json | Output file for metrics |\n| `--load_in_8bit` | False | 8-bit quantization |\n| `--load_in_4bit` | False | 4-bit quantization |\n| `--trust_remote_code` | False | Allow custom model code |\n| `--precision` | fp32 | Model precision (fp32\u002Ffp16\u002Fbf16) |\n\n## Hardware Requirements\n\n| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |\n|------------|-------------|--------------|-------------------------|\n| 7B | 14GB | 6GB | ~30 min (A100) |\n| 13B | 26GB | 10GB | ~1 hour (A100) |\n| 34B | 68GB | 20GB | ~2 hours (A100) |\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\n- **Documentation**: https:\u002F\u002Fgithub.com\u002Fbigcode-project\u002Fbigcode-evaluation-harness\u002Ftree\u002Fmain\u002Fdocs\n- **BigCode Leaderboard**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbigcode\u002Fbigcode-models-leaderboard\n- **HumanEval Dataset**: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fopenai_humaneval\n- **MultiPL-E**: https:\u002F\u002Fgithub.com\u002Fnuprl\u002FMultiPL-E\n",{"name":5,"tags":49,"author":53,"license":15,"version":54,"category":55,"description":7,"dependencies":56},[40,39,38,37,36,50,51,52,27],"Pass@k","BigCode","Benchmarking","Synthetic Sciences","1.0.0","ml-training",[57,58,59,60],"bigcode-evaluation-harness","transformers>=4.25.1","accelerate>=0.13.2","datasets>=2.6.1","main","4aa39efd1cdab0b4657ecee16c92a92011a5b10f","openscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fbigcode-evaluation-harness","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub","https:\u002F\u002Fgithub.com\u002Ffadrienne","2026-10-28 00:43:09",{"source":8,"skillName":5,"commands":68,"prompt":78,"zipAvailable":79},[69,72,75],{"tool":70,"command":71},"npx","npx skills add fadrienne\u002FMyGitHub --skill evaluating-code-models",{"tool":73,"command":74},"bunx","bunx skills add fadrienne\u002FMyGitHub --skill evaluating-code-models",{"tool":76,"command":77},"pnpm","pnpm dlx skills add fadrienne\u002FMyGitHub --skill evaluating-code-models","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fbigcode-evaluation-harness\u002FSKILL.md",true,[81,87,91],{"path":82,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fbenchmarks.md","REFERENCE","参考文档",9168,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fbigcode-evaluation-harness\u002Freferences\u002Fbenchmarks.md",{"path":88,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fcustom-tasks.md",10378,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fbigcode-evaluation-harness\u002Freferences\u002Fcustom-tasks.md",{"path":92,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Fissues.md",8085,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fbigcode-evaluation-harness\u002Freferences\u002Fissues.md",[96,127,153,185,212,236],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":107,"publishedAt":44},"01M7167B28PK20HP7R7ZED2DY0","statistical-power","Sample size calculation, power analysis, and effect size conversion for scientific studies. Use BEFORE data collection (a priori) to determine required N, or after (post-hoc) to compute achieved power. Covers parametric tests (t-test, ANOVA, chi-square, proportions, correlation, regression), simulation-based power for complex\u002Fclustered\u002Fnested designs, and power curves. Use this skill when someone asks 'how many samples do I need', 'what is my power', 'what effect size can I detect', or any variant. Pairs with \u002Fexperimental-design (choose design first, then compute power) and \u002Fstatistical-analysis (analyze data after collection).","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-power\u002FSKILL.md",[102],{"categoryNo":103,"categoryName":104,"taxonomyType":105,"taxonomyTypeDesc":106},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[108,109,110,111,112,113,114,115,28,116,117,118,119,120,121,122,123,124,125,126],"power","statistical power","effect","scipy","regression","choose","design","uv","Bash","collection","analysis","calculation","ANOVA","conversion","analyze","t-test","chi-square","simulation","sample",{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":136,"publishedAt":44},"01M71677EBCM8EPCE9BZCK8505","statistical-analysis","Guided statistical analysis with test selection and reporting. Use when you need help choosing appropriate tests for your data, assumption checking, power analysis, and APA-formatted results. Best for academic research reporting, test selection guidance. For implementing specific models programmatically use statsmodels.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-analysis\u002FSKILL.md",[133],{"categoryNo":134,"categoryName":135,"taxonomyType":105,"taxonomyTypeDesc":106},"01JBSEEDCAT000000000000008","测试与质量",[118,137,108,138,139,140,28,141,142,143,144,145,146,147,148,149,150,151,26,152],"statistical analysis","academic research","implementing","selection","you","reporting","data","for","tests","use","with","test","research","results","statsmodels","help",{"skillNo":154,"name":155,"nameZh":6,"description":156,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":157,"categories":158,"keywords":166,"publishedAt":44},"01M71673R93XAHFH81DBNVNG7H","scientific-writing","Core skill for the deep research and writing tool. Write scientific manuscripts in full paragraphs (never bullet points). Use two-stage process with (1) section outlines with key points using research-lookup then (2) convert to flowing prose. IMRAD structure, citations (APA\u002FAMA\u002FVancouver), figures\u002Ftables, reporting guidelines (CONSORT\u002FSTROBE\u002FPRISMA), for research papers and journal submissions.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-writing\u002FSKILL.md",[159,162,163],{"categoryNo":160,"categoryName":161,"taxonomyType":105,"taxonomyTypeDesc":106},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":103,"categoryName":104,"taxonomyType":105,"taxonomyTypeDesc":106},{"categoryNo":164,"categoryName":165,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000036","内容创作者",[167,168,169,170,171,172,173,174,175,176,116,177,178,179,180,181,182,183,142,184],"writing","scientific","scientific writing","Citations","deep","deep research","never","papers","journal","false","api","prisma","structure","openrouter","Outlines","core skill","read","features",{"skillNo":186,"name":187,"nameZh":6,"description":188,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":189,"categories":190,"keywords":195,"publishedAt":44},"01M7166ZPKXBF4EBPHVTJPQBB1","scientific-visualization","Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib\u002Fseaborn\u002Fplotly with publication styles. For quick exploration use seaborn or plotly directly.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-visualization\u002FSKILL.md",[191,194],{"categoryNo":192,"categoryName":193,"taxonomyType":105,"taxonomyTypeDesc":106},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":103,"categoryName":104,"taxonomyType":105,"taxonomyTypeDesc":106},[196,168,197,198,199,200,201,202,203,175,28,204,205,206,207,208,209,210,144,211],"visualization","scientific visualization","error","ready","quick","exploration","meta-skill","Meta","safe","plotly","annotations","formatting","Publication","figures","panel","submission",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":216,"categories":217,"keywords":222,"publishedAt":44},"01M7166VXJWQSSC3RQZP60WBGR","research-code","Debug, profile, review, and restructure Python research and analysis code. Use when analysis scripts crash, produce wrong results, run too slowly, or need a correctness review. Covers: reading tracebacks and diagnosing root causes, hypothesis-driven debugging, reviewing code for common statistical\u002Fnumerical errors, profiling data pipelines, improving reproducibility (seeds, environments, determinism), and structuring analysis scripts for sharing. Distinct from \u002Fexploratory-data-analysis (which understands data) and \u002Fstatistical-analysis (which chooses and runs tests) — this skill focuses on the code itself.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fresearch-code\u002FSKILL.md",[218,221],{"categoryNo":219,"categoryName":220,"taxonomyType":105,"taxonomyTypeDesc":106},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[25,223,149,224,225,226,227,115,228,28,116,118,229,230,231,232,233,234,183,235],"research code","which","memory","data-analysis","errors","debugging","common","review","data pipelines","run","scripts","line","write",{"skillNo":237,"name":238,"nameZh":6,"description":239,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":240,"licenseFileUrl":6,"sourceHtmlUrl":241,"categories":242,"keywords":249,"publishedAt":44},"01M7166CF5HSR7SVZYZFWTDX55","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fmarkdown-mermaid-writing\u002FSKILL.md",[243,246,247,248],{"categoryNo":244,"categoryName":245,"taxonomyType":105,"taxonomyTypeDesc":106},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":160,"categoryName":161,"taxonomyType":105,"taxonomyTypeDesc":106},{"categoryNo":103,"categoryName":104,"taxonomyType":105,"taxonomyTypeDesc":106},{"categoryNo":164,"categoryName":165,"taxonomyType":21,"taxonomyTypeDesc":22},[167,250,251,252,253,254,196,33,255,28,116,118,256,257,258,259,260,261,168,183],"markdown mermaid writing","mermaid","markdown","community","diagrams","default","team","guides","text","documentation","references","diagram"]