[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EB5C34JRHMJ7TA9CWMSSJ8":3,"skill-same:01M6EB5C34JRHMJ7TA9CWMSSJ8":116},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":26,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":60,"commitSha":61,"skillDirectory":62,"forksCount":63,"repositoryHtmlUrl":64,"repositoryHomepageUrl":6,"ownerHtmlUrl":65,"ownerBlogUrl":6,"ownerDescription":66,"updatedAt":67,"install":68,"resources":81},"01M6EB5C34JRHMJ7TA9CWMSSJ8","nemo-evaluator-sdk",null,"Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.","luokai0\u002Fai-agent-skills-by-luo-kai","luokai0","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F266954973?v=4","01M6E16VNT42JB8B3X3254QWJ7","luo kai","01M6E16VTFVP6S5XJF1V1MPV90",14,"MIT","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46],"nemo evaluator sdk","sdk","evaluator","NeMo","Multi-Backend","platform","local","across","launcher","Safety","Reproducible","Slurm","llms","MMLU","NVIDIA","HumanEval","Evaluation","benchmarking","architecture","docker","2026-11-10 10:12:30",false,"---\nname: nemo-evaluator-sdk\ndescription: Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.\nversion: 1.0.0\nauthor: luo-kai\nlicense: MIT\ntags: [Evaluation, NeMo, NVIDIA, Benchmarking, MMLU, HumanEval, Multi-Backend, Slurm, Docker, Reproducible, Enterprise]\ndependencies: [nemo-evaluator-launcher>=0.1.25, docker]\n---\n\n# NeMo Evaluator SDK - Enterprise LLM Benchmarking\n\n## Quick Start\n\nNeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).\n\n**Installation**:\n```bash\npip install nemo-evaluator-launcher\n```\n\n**Set API key and run evaluation**:\n```bash\nexport NGC_API_KEY=nvapi-your-key-here\n\n# Create minimal config\ncat > config.yaml \u003C\u003C 'EOF'\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n\nevaluation:\n  tasks:\n    - name: ifeval\nEOF\n\n# Run evaluation\nnemo-evaluator-launcher run --config-dir . --config-name config\n```\n\n**View available tasks**:\n```bash\nnemo-evaluator-launcher ls tasks\n```\n\n## Common Workflows\n\n### Workflow 1: Evaluate Model on Standard Benchmarks\n\nRun core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint.\n\n**Checklist**:\n```\nStandard Evaluation:\n- [ ] Step 1: Configure API endpoint\n- [ ] Step 2: Select benchmarks\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Check results\n```\n\n**Step 1: Configure API endpoint**\n\n```yaml\n# config.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n```\n\nFor self-hosted endpoints (vLLM, TRT-LLM):\n```yaml\ntarget:\n  api_endpoint:\n    model_id: my-model\n    url: http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: \"\"  # No key needed for local\n```\n\n**Step 2: Select benchmarks**\n\nAdd tasks to your config:\n```yaml\nevaluation:\n  tasks:\n    - name: ifeval           # Instruction following\n    - name: gpqa_diamond     # Graduate-level QA\n      env_vars:\n        HF_TOKEN: HF_TOKEN   # Some tasks need HF token\n    - name: gsm8k_cot_instruct  # Math reasoning\n    - name: humaneval        # Code generation\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Run with config file\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config\n\n# Override output directory\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o execution.output_dir=.\u002Fmy_results\n\n# Limit samples for quick testing\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=10\n```\n\n**Step 4: Check results**\n\n```bash\n# Check job status\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# List all runs\nnemo-evaluator-launcher ls runs\n\n# View results\ncat results\u002F\u003Cinvocation_id>\u002F\u003Ctask>\u002Fartifacts\u002Fresults.yml\n```\n\n### Workflow 2: Run Evaluation on Slurm HPC Cluster\n\nExecute large-scale evaluation on HPC infrastructure.\n\n**Checklist**:\n```\nSlurm Evaluation:\n- [ ] Step 1: Configure Slurm settings\n- [ ] Step 2: Set up model deployment\n- [ ] Step 3: Launch evaluation\n- [ ] Step 4: Monitor job status\n```\n\n**Step 1: Configure Slurm settings**\n\n```yaml\n# slurm_config.yaml\ndefaults:\n  - execution: slurm\n  - deployment: vllm\n  - _self_\n\nexecution:\n  hostname: cluster.example.com\n  account: my_slurm_account\n  partition: gpu\n  output_dir: \u002Fshared\u002Fresults\n  walltime: \"04:00:00\"\n  nodes: 1\n  gpus_per_node: 8\n```\n\n**Step 2: Set up model deployment**\n\n```yaml\ndeployment:\n  checkpoint_path: \u002Fshared\u002Fmodels\u002Fllama-3.1-8b\n  tensor_parallel_size: 2\n  data_parallel_size: 4\n  max_model_len: 4096\n\ntarget:\n  api_endpoint:\n    model_id: llama-3.1-8b\n    # URL auto-generated by deployment\n```\n\n**Step 3: Launch evaluation**\n\n```bash\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name slurm_config\n```\n\n**Step 4: Monitor job status**\n\n```bash\n# Check status (queries sacct)\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# View detailed info\nnemo-evaluator-launcher info \u003Cinvocation_id>\n\n# Kill if needed\nnemo-evaluator-launcher kill \u003Cinvocation_id>\n```\n\n### Workflow 3: Compare Multiple Models\n\nBenchmark multiple models on the same tasks for comparison.\n\n**Checklist**:\n```\nModel Comparison:\n- [ ] Step 1: Create base config\n- [ ] Step 2: Run evaluations with overrides\n- [ ] Step 3: Export and compare results\n```\n\n**Step 1: Create base config**\n\n```yaml\n# base_eval.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fcomparison_results\n\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.01\n        parallelism: 4\n  tasks:\n    - name: mmlu_pro\n    - name: gsm8k_cot_instruct\n    - name: ifeval\n```\n\n**Step 2: Run evaluations with model overrides**\n\n```bash\n# Evaluate Llama 3.1 8B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=meta\u002Fllama-3.1-8b-instruct \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\n# Evaluate Mistral 7B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=mistralai\u002Fmistral-7b-instruct-v0.3 \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n```\n\n**Step 3: Export and compare**\n\n```bash\n# Export to MLflow\nnemo-evaluator-launcher export \u003Cinvocation_id_1> --dest mlflow\nnemo-evaluator-launcher export \u003Cinvocation_id_2> --dest mlflow\n\n# Export to local JSON\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest local --format json\n\n# Export to Weights & Biases\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest wandb\n```\n\n### Workflow 4: Safety and Vision-Language Evaluation\n\nEvaluate models on safety benchmarks and VLM tasks.\n\n**Checklist**:\n```\nSafety\u002FVLM Evaluation:\n- [ ] Step 1: Configure safety tasks\n- [ ] Step 2: Set up VLM tasks (if applicable)\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Configure safety tasks**\n\n```yaml\nevaluation:\n  tasks:\n    - name: aegis              # Safety harness\n    - name: wildguard          # Safety classification\n    - name: garak              # Security probing\n```\n\n**Step 2: Configure VLM tasks**\n\n```yaml\n# For vision-language models\ntarget:\n  api_endpoint:\n    type: vlm  # Vision-language endpoint\n    model_id: nvidia\u002Fllama-3.2-90b-vision-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\nevaluation:\n  tasks:\n    - name: ocrbench           # OCR evaluation\n    - name: chartqa            # Chart understanding\n    - name: mmmu               # Multimodal understanding\n```\n\n## When to Use vs Alternatives\n\n**Use NeMo Evaluator when:**\n- Need **100+ benchmarks** from 18+ harnesses in one platform\n- Running evaluations on **Slurm HPC clusters** or cloud\n- Requiring **reproducible** containerized evaluation\n- Evaluating against **OpenAI-compatible APIs** (vLLM, TRT-LLM, NIMs)\n- Need **enterprise-grade** evaluation with result export (MLflow, W&B)\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: Simpler setup for quick local evaluation\n- **bigcode-evaluation-harness**: Focused only on code benchmarks\n- **HELM**: Stanford's broader evaluation (fairness, efficiency)\n- **Custom scripts**: Highly specialized domain evaluation\n\n## Supported Harnesses and Tasks\n\n| Harness | Task Count | Categories |\n|---------|-----------|------------|\n| `lm-evaluation-harness` | 60+ | MMLU, GSM8K, HellaSwag, ARC |\n| `simple-evals` | 20+ | GPQA, MATH, AIME |\n| `bigcode-evaluation-harness` | 25+ | HumanEval, MBPP, MultiPL-E |\n| `safety-harness` | 3 | Aegis, WildGuard |\n| `garak` | 1 | Security probing |\n| `vlmevalkit` | 6+ | OCRBench, ChartQA, MMMU |\n| `bfcl` | 6 | Function calling v2\u002Fv3 |\n| `mtbench` | 2 | Multi-turn conversation |\n| `livecodebench` | 10+ | Live coding evaluation |\n| `helm` | 15 | Medical domain |\n| `nemo-skills` | 8 | Math, science, agentic |\n\n## Common Issues\n\n**Issue: Container pull fails**\n\nEnsure NGC credentials are configured:\n```bash\ndocker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY\n```\n\n**Issue: Task requires environment variable**\n\nSome tasks need HF_TOKEN or JUDGE_API_KEY:\n```yaml\nevaluation:\n  tasks:\n    - name: gpqa_diamond\n      env_vars:\n        HF_TOKEN: HF_TOKEN  # Maps env var name to env var\n```\n\n**Issue: Evaluation timeout**\n\nIncrease parallelism or reduce samples:\n```bash\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100\n```\n\n**Issue: Slurm job not starting**\n\nCheck Slurm account and partition:\n```yaml\nexecution:\n  account: correct_account\n  partition: gpu\n  qos: normal  # May need specific QOS\n```\n\n**Issue: Different results than expected**\n\nVerify configuration matches reported settings:\n```yaml\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.0  # Deterministic\n        num_fewshot: 5    # Check paper's fewshot count\n```\n\n## CLI Reference\n\n| Command | Description |\n|---------|-------------|\n| `run` | Execute evaluation with config |\n| `status \u003Cid>` | Check job status |\n| `info \u003Cid>` | View detailed job info |\n| `ls tasks` | List available benchmarks |\n| `ls runs` | List all invocations |\n| `export \u003Cid>` | Export results (mlflow\u002Fwandb\u002Flocal) |\n| `kill \u003Cid>` | Terminate running job |\n\n## Configuration Override Examples\n\n```bash\n# Override model endpoint\n-o target.api_endpoint.model_id=my-model\n-o target.api_endpoint.url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n\n# Add evaluation parameters\n-o +evaluation.nemo_evaluator_config.config.params.temperature=0.5\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=50\n\n# Change execution settings\n-o execution.output_dir=\u002Fcustom\u002Fpath\n-o execution.mode=parallel\n\n# Dynamically set tasks\n-o 'evaluation.tasks=[{name: ifeval}, {name: gsm8k}]'\n```\n\n## Python API Usage\n\nFor programmatic evaluation without the CLI:\n\n```python\nfrom nemo_evaluator.core.evaluate import evaluate\nfrom nemo_evaluator.api.api_dataclasses import (\n    EvaluationConfig,\n    EvaluationTarget,\n    ApiEndpoint,\n    EndpointType,\n    ConfigParams\n)\n\n# Configure evaluation\neval_config = EvaluationConfig(\n    type=\"mmlu_pro\",\n    output_dir=\".\u002Fresults\",\n    params=ConfigParams(\n        limit_samples=10,\n        temperature=0.0,\n        max_new_tokens=1024,\n        parallelism=4\n    )\n)\n\n# Configure target endpoint\ntarget_config = EvaluationTarget(\n    api_endpoint=ApiEndpoint(\n        model_id=\"meta\u002Fllama-3.1-8b-instruct\",\n        url=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\",\n        type=EndpointType.CHAT,\n        api_key=\"nvapi-your-key-here\"\n    )\n)\n\n# Run evaluation\nresult = evaluate(eval_cfg=eval_config, target_cfg=target_config)\n```\n\n## Advanced Topics\n\n**Multi-backend execution**: See [references\u002Fexecution-backends.md](references\u002Fexecution-backends.md)\n**Configuration deep-dive**: See [references\u002Fconfiguration.md](references\u002Fconfiguration.md)\n**Adapter and interceptor system**: See [references\u002Fadapter-system.md](references\u002Fadapter-system.md)\n**Custom benchmark integration**: See [references\u002Fcustom-benchmarks.md](references\u002Fcustom-benchmarks.md)\n\n## Requirements\n\n- **Python**: 3.10-3.13\n- **Docker**: Required for local execution\n- **NGC API Key**: For pulling containers and using NVIDIA Build\n- **HF_TOKEN**: Required for some benchmarks (GPQA, MMLU)\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\n- **NGC Containers**: nvcr.io\u002Fnvidia\u002Feval-factory\u002F\n- **NVIDIA Build**: https:\u002F\u002Fbuild.nvidia.com (free hosted models)\n- **Documentation**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Ftree\u002Fmain\u002Fdocs\n","# NeMo Evaluator SDK - Enterprise LLM Benchmarking\n\n## Quick Start\n\nNeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).\n\n**Installation**:\n```bash\npip install nemo-evaluator-launcher\n```\n\n**Set API key and run evaluation**:\n```bash\nexport NGC_API_KEY=nvapi-your-key-here\n\n# Create minimal config\ncat > config.yaml \u003C\u003C 'EOF'\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n\nevaluation:\n  tasks:\n    - name: ifeval\nEOF\n\n# Run evaluation\nnemo-evaluator-launcher run --config-dir . --config-name config\n```\n\n**View available tasks**:\n```bash\nnemo-evaluator-launcher ls tasks\n```\n\n## Common Workflows\n\n### Workflow 1: Evaluate Model on Standard Benchmarks\n\nRun core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint.\n\n**Checklist**:\n```\nStandard Evaluation:\n- [ ] Step 1: Configure API endpoint\n- [ ] Step 2: Select benchmarks\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Check results\n```\n\n**Step 1: Configure API endpoint**\n\n```yaml\n# config.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n```\n\nFor self-hosted endpoints (vLLM, TRT-LLM):\n```yaml\ntarget:\n  api_endpoint:\n    model_id: my-model\n    url: http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: \"\"  # No key needed for local\n```\n\n**Step 2: Select benchmarks**\n\nAdd tasks to your config:\n```yaml\nevaluation:\n  tasks:\n    - name: ifeval           # Instruction following\n    - name: gpqa_diamond     # Graduate-level QA\n      env_vars:\n        HF_TOKEN: HF_TOKEN   # Some tasks need HF token\n    - name: gsm8k_cot_instruct  # Math reasoning\n    - name: humaneval        # Code generation\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Run with config file\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config\n\n# Override output directory\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o execution.output_dir=.\u002Fmy_results\n\n# Limit samples for quick testing\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=10\n```\n\n**Step 4: Check results**\n\n```bash\n# Check job status\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# List all runs\nnemo-evaluator-launcher ls runs\n\n# View results\ncat results\u002F\u003Cinvocation_id>\u002F\u003Ctask>\u002Fartifacts\u002Fresults.yml\n```\n\n### Workflow 2: Run Evaluation on Slurm HPC Cluster\n\nExecute large-scale evaluation on HPC infrastructure.\n\n**Checklist**:\n```\nSlurm Evaluation:\n- [ ] Step 1: Configure Slurm settings\n- [ ] Step 2: Set up model deployment\n- [ ] Step 3: Launch evaluation\n- [ ] Step 4: Monitor job status\n```\n\n**Step 1: Configure Slurm settings**\n\n```yaml\n# slurm_config.yaml\ndefaults:\n  - execution: slurm\n  - deployment: vllm\n  - _self_\n\nexecution:\n  hostname: cluster.example.com\n  account: my_slurm_account\n  partition: gpu\n  output_dir: \u002Fshared\u002Fresults\n  walltime: \"04:00:00\"\n  nodes: 1\n  gpus_per_node: 8\n```\n\n**Step 2: Set up model deployment**\n\n```yaml\ndeployment:\n  checkpoint_path: \u002Fshared\u002Fmodels\u002Fllama-3.1-8b\n  tensor_parallel_size: 2\n  data_parallel_size: 4\n  max_model_len: 4096\n\ntarget:\n  api_endpoint:\n    model_id: llama-3.1-8b\n    # URL auto-generated by deployment\n```\n\n**Step 3: Launch evaluation**\n\n```bash\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name slurm_config\n```\n\n**Step 4: Monitor job status**\n\n```bash\n# Check status (queries sacct)\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# View detailed info\nnemo-evaluator-launcher info \u003Cinvocation_id>\n\n# Kill if needed\nnemo-evaluator-launcher kill \u003Cinvocation_id>\n```\n\n### Workflow 3: Compare Multiple Models\n\nBenchmark multiple models on the same tasks for comparison.\n\n**Checklist**:\n```\nModel Comparison:\n- [ ] Step 1: Create base config\n- [ ] Step 2: Run evaluations with overrides\n- [ ] Step 3: Export and compare results\n```\n\n**Step 1: Create base config**\n\n```yaml\n# base_eval.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fcomparison_results\n\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.01\n        parallelism: 4\n  tasks:\n    - name: mmlu_pro\n    - name: gsm8k_cot_instruct\n    - name: ifeval\n```\n\n**Step 2: Run evaluations with model overrides**\n\n```bash\n# Evaluate Llama 3.1 8B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=meta\u002Fllama-3.1-8b-instruct \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\n# Evaluate Mistral 7B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=mistralai\u002Fmistral-7b-instruct-v0.3 \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n```\n\n**Step 3: Export and compare**\n\n```bash\n# Export to MLflow\nnemo-evaluator-launcher export \u003Cinvocation_id_1> --dest mlflow\nnemo-evaluator-launcher export \u003Cinvocation_id_2> --dest mlflow\n\n# Export to local JSON\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest local --format json\n\n# Export to Weights & Biases\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest wandb\n```\n\n### Workflow 4: Safety and Vision-Language Evaluation\n\nEvaluate models on safety benchmarks and VLM tasks.\n\n**Checklist**:\n```\nSafety\u002FVLM Evaluation:\n- [ ] Step 1: Configure safety tasks\n- [ ] Step 2: Set up VLM tasks (if applicable)\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Configure safety tasks**\n\n```yaml\nevaluation:\n  tasks:\n    - name: aegis              # Safety harness\n    - name: wildguard          # Safety classification\n    - name: garak              # Security probing\n```\n\n**Step 2: Configure VLM tasks**\n\n```yaml\n# For vision-language models\ntarget:\n  api_endpoint:\n    type: vlm  # Vision-language endpoint\n    model_id: nvidia\u002Fllama-3.2-90b-vision-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\nevaluation:\n  tasks:\n    - name: ocrbench           # OCR evaluation\n    - name: chartqa            # Chart understanding\n    - name: mmmu               # Multimodal understanding\n```\n\n## When to Use vs Alternatives\n\n**Use NeMo Evaluator when:**\n- Need **100+ benchmarks** from 18+ harnesses in one platform\n- Running evaluations on **Slurm HPC clusters** or cloud\n- Requiring **reproducible** containerized evaluation\n- Evaluating against **OpenAI-compatible APIs** (vLLM, TRT-LLM, NIMs)\n- Need **enterprise-grade** evaluation with result export (MLflow, W&B)\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: Simpler setup for quick local evaluation\n- **bigcode-evaluation-harness**: Focused only on code benchmarks\n- **HELM**: Stanford's broader evaluation (fairness, efficiency)\n- **Custom scripts**: Highly specialized domain evaluation\n\n## Supported Harnesses and Tasks\n\n| Harness | Task Count | Categories |\n|---------|-----------|------------|\n| `lm-evaluation-harness` | 60+ | MMLU, GSM8K, HellaSwag, ARC |\n| `simple-evals` | 20+ | GPQA, MATH, AIME |\n| `bigcode-evaluation-harness` | 25+ | HumanEval, MBPP, MultiPL-E |\n| `safety-harness` | 3 | Aegis, WildGuard |\n| `garak` | 1 | Security probing |\n| `vlmevalkit` | 6+ | OCRBench, ChartQA, MMMU |\n| `bfcl` | 6 | Function calling v2\u002Fv3 |\n| `mtbench` | 2 | Multi-turn conversation |\n| `livecodebench` | 10+ | Live coding evaluation |\n| `helm` | 15 | Medical domain |\n| `nemo-skills` | 8 | Math, science, agentic |\n\n## Common Issues\n\n**Issue: Container pull fails**\n\nEnsure NGC credentials are configured:\n```bash\ndocker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY\n```\n\n**Issue: Task requires environment variable**\n\nSome tasks need HF_TOKEN or JUDGE_API_KEY:\n```yaml\nevaluation:\n  tasks:\n    - name: gpqa_diamond\n      env_vars:\n        HF_TOKEN: HF_TOKEN  # Maps env var name to env var\n```\n\n**Issue: Evaluation timeout**\n\nIncrease parallelism or reduce samples:\n```bash\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100\n```\n\n**Issue: Slurm job not starting**\n\nCheck Slurm account and partition:\n```yaml\nexecution:\n  account: correct_account\n  partition: gpu\n  qos: normal  # May need specific QOS\n```\n\n**Issue: Different results than expected**\n\nVerify configuration matches reported settings:\n```yaml\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.0  # Deterministic\n        num_fewshot: 5    # Check paper's fewshot count\n```\n\n## CLI Reference\n\n| Command | Description |\n|---------|-------------|\n| `run` | Execute evaluation with config |\n| `status \u003Cid>` | Check job status |\n| `info \u003Cid>` | View detailed job info |\n| `ls tasks` | List available benchmarks |\n| `ls runs` | List all invocations |\n| `export \u003Cid>` | Export results (mlflow\u002Fwandb\u002Flocal) |\n| `kill \u003Cid>` | Terminate running job |\n\n## Configuration Override Examples\n\n```bash\n# Override model endpoint\n-o target.api_endpoint.model_id=my-model\n-o target.api_endpoint.url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n\n# Add evaluation parameters\n-o +evaluation.nemo_evaluator_config.config.params.temperature=0.5\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=50\n\n# Change execution settings\n-o execution.output_dir=\u002Fcustom\u002Fpath\n-o execution.mode=parallel\n\n# Dynamically set tasks\n-o 'evaluation.tasks=[{name: ifeval}, {name: gsm8k}]'\n```\n\n## Python API Usage\n\nFor programmatic evaluation without the CLI:\n\n```python\nfrom nemo_evaluator.core.evaluate import evaluate\nfrom nemo_evaluator.api.api_dataclasses import (\n    EvaluationConfig,\n    EvaluationTarget,\n    ApiEndpoint,\n    EndpointType,\n    ConfigParams\n)\n\n# Configure evaluation\neval_config = EvaluationConfig(\n    type=\"mmlu_pro\",\n    output_dir=\".\u002Fresults\",\n    params=ConfigParams(\n        limit_samples=10,\n        temperature=0.0,\n        max_new_tokens=1024,\n        parallelism=4\n    )\n)\n\n# Configure target endpoint\ntarget_config = EvaluationTarget(\n    api_endpoint=ApiEndpoint(\n        model_id=\"meta\u002Fllama-3.1-8b-instruct\",\n        url=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\",\n        type=EndpointType.CHAT,\n        api_key=\"nvapi-your-key-here\"\n    )\n)\n\n# Run evaluation\nresult = evaluate(eval_cfg=eval_config, target_cfg=target_config)\n```\n\n## Advanced Topics\n\n**Multi-backend execution**: See [references\u002Fexecution-backends.md](references\u002Fexecution-backends.md)\n**Configuration deep-dive**: See [references\u002Fconfiguration.md](references\u002Fconfiguration.md)\n**Adapter and interceptor system**: See [references\u002Fadapter-system.md](references\u002Fadapter-system.md)\n**Custom benchmark integration**: See [references\u002Fcustom-benchmarks.md](references\u002Fcustom-benchmarks.md)\n\n## Requirements\n\n- **Python**: 3.10-3.13\n- **Docker**: Required for local execution\n- **NGC API Key**: For pulling containers and using NVIDIA Build\n- **HF_TOKEN**: Required for some benchmarks (GPQA, MMLU)\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\n- **NGC Containers**: nvcr.io\u002Fnvidia\u002Feval-factory\u002F\n- **NVIDIA Build**: https:\u002F\u002Fbuild.nvidia.com (free hosted models)\n- **Documentation**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Ftree\u002Fmain\u002Fdocs\n",{"name":5,"tags":52,"author":56,"license":15,"version":57,"description":7,"dependencies":58},[43,30,41,53,40,42,31,38,54,37,55],"Benchmarking","Docker","Enterprise","luo-kai","1.0.0",[59,46],"nemo-evaluator-launcher>=0.1.25","main","4d2fb89cdfb75f3f6fd3c6c5ec9ad90ea129eb3c","ai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator",1,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai","https:\u002F\u002Fgithub.com\u002Fluokai0","a solo builder.","2026-05-06 04:29:04",{"source":8,"skillName":5,"commands":69,"prompt":79,"zipAvailable":80},[70,73,76],{"tool":71,"command":72},"npx","npx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill nemo-evaluator-sdk",{"tool":74,"command":75},"bunx","bunx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill nemo-evaluator-sdk",{"tool":77,"command":78},"pnpm","pnpm dlx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill nemo-evaluator-sdk","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator\u002FSKILL.md",true,[82,88,94,100,104,108,112],{"path":83,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":86,"sourceHtmlUrl":87},"customize.toml","OTHER","其他",1654,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.iflow\u002Fskills\u002Fbmad-cis-design-thinking\u002Fcustomize.toml",{"path":89,"resourceType":90,"resourceTypeDesc":91,"sizeBytes":92,"sourceHtmlUrl":93},"design-methods.csv","ASSET","资产文件",6412,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.iflow\u002Fskills\u002Fbmad-cis-design-thinking\u002Fdesign-methods.csv",{"path":95,"resourceType":96,"resourceTypeDesc":97,"sizeBytes":98,"sourceHtmlUrl":99},"references\u002Fadapter-system.md","REFERENCE","参考文档",9468,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator\u002Freferences\u002Fadapter-system.md",{"path":101,"resourceType":96,"resourceTypeDesc":97,"sizeBytes":102,"sourceHtmlUrl":103},"references\u002Fconfiguration.md",9567,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator\u002Freferences\u002Fconfiguration.md",{"path":105,"resourceType":96,"resourceTypeDesc":97,"sizeBytes":106,"sourceHtmlUrl":107},"references\u002Fcustom-benchmarks.md",7101,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator\u002Freferences\u002Fcustom-benchmarks.md",{"path":109,"resourceType":96,"resourceTypeDesc":97,"sizeBytes":110,"sourceHtmlUrl":111},"references\u002Fexecution-backends.md",9351,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fnemo-evaluator\u002Freferences\u002Fexecution-backends.md",{"path":113,"resourceType":96,"resourceTypeDesc":97,"sizeBytes":114,"sourceHtmlUrl":115},"template.md",1278,"https:\u002F\u002Fgithub.com\u002FdevelopAgent\u002Fbmad-start\u002Fblob\u002Fmain\u002F.iflow\u002Fskills\u002Fbmad-cis-design-thinking\u002Ftemplate.md",[117,144,161,189,222,257],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":47},"01M6F2SKTE208174C7S7JJCZC7","write-contracts","Generates secure Aptos Move V2 smart contracts with Object model, Digital Asset integration, security patterns, and storage type guidance. Includes comprehensive storage decision framework for optimal data structure selection. Triggers on: 'write contract', 'create NFT collection', 'build marketplace', 'implement minting', 'generate Move module', 'create token contract', 'build DAO', 'implement staking'. Ask storage questions when: 'store', 'track', 'registry', 'mapping', 'list', 'collection'.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fwrite-contracts\u002FSKILL.md",[],[124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143],"write","contracts","write contracts","storage","secure","create","generate","collection","model","decision","ask","smart-contracts","selection","objects","structure","nft","token","aptos","critical","move",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":148,"categories":149,"keywords":150,"publishedAt":47},"01M6F2RFXJ6T9MKW0A7N7GM45B","web-search","通用网络搜索技能，支持多引擎搜索（百度、必应、DuckDuckGo），无需API密钥即可获取实时信息","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fweb-search-ex-skill\u002FSKILL.md",[],[151,152,153,154,155,156,157,158,159,160],"web","search","web search","搜索","baidu","bing","duckduckgo","百度","多引擎","internet",{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":170,"publishedAt":47},"01M6F2K968G1FZEZFPW2E6B222","solo-build","Execute implementation plan tasks with TDD workflow, auto-commit, and phase gates. Use when user says \"build it\", \"start building\", \"execute plan\", \"implement tasks\", \"ship it\", or references a track ID. Do NOT use for planning (use \u002Fplan) or scaffolding (use \u002Fscaffold).","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fsolo-build\u002FSKILL.md",[167],{"categoryNo":168,"categoryName":169,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[171,172,173,174,175,176,177,178,152,179,180,181,124,182,183,184,185,186,187,188],"build","solo","solo build","MCP","workflow","commit","tasks","code","Bash","plan","read","auto","references","context7","query","scaffolding","ship it","project",{"skillNo":190,"name":191,"nameZh":6,"description":192,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":193,"categories":194,"keywords":201,"publishedAt":47},"01M6F2GKN39SGHSPWYB3662YC8","react-email","Use when building HTML email templates with React components, adding a visual email editor to an application using the React Email visual editor, rendering emails to HTML, or sending emails with Resend. Covers welcome emails, password resets, notifications, order confirmations, newsletters, transactional emails, and the embeddable email editor component.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Freact-email-skills\u002FSKILL.md",[195,198],{"categoryNo":196,"categoryName":197,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":199,"categoryName":200,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000009","前端开发",[202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221],"react","email","react email","github","editor","components","react components","component","welcome","https","notifications","password","node","use","html","resend","visual","emails","rendering","with",{"skillNo":223,"name":224,"nameZh":6,"description":225,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":226,"categories":227,"keywords":238,"publishedAt":47},"01M6F2FBVGVDFSG8SSFJGXA40C","APITester Agent-Driven API Testing","Test API endpoints and document responses. Define tests in plain English, run them, get formatted results. Agent-driven Postman alternative.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fopenclaw-api-tester\u002FSKILL.md",[228,231,232,233],{"categoryNo":229,"categoryName":230,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":168,"categoryName":169,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":234,"categoryName":235,"taxonomyType":236,"taxonomyTypeDesc":237},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256],"testing","api","api testing","driven","APITester Agent Driven API Testing","test","run","postman","results","plain english","tester","endpoints","document","get","define","tests","english","api-tester",{"skillNo":258,"name":259,"nameZh":6,"description":260,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":261,"categories":262,"keywords":263,"publishedAt":47},"01M6F273Y3NGAPVJZ608BC0G9G","hubspot-implementation-plan","Generate a phased implementation plan from a HubSpot audit report. Creates prioritized, sequenced cleanup processes with effort estimates, dependencies, and automation feasibility. Use after running \u002Fhubspot-audit.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fhubspot-implementation-plan\u002FSKILL.md",[],[180,264,265,266,267,268,269,130,270,271,272,273,274,215,275,276,221,277],"implementation","hubspot","hubspot implementation plan","automation","feasibility","cleanup","planning","report","audit","audit report","phased implementation","dependencies","processes","hubspot audit"]