[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M77XQ37PPZ59FS22HGEZRZPY":3,"skill-same:01M77XQ37PPZ59FS22HGEZRZPY":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":14,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M77XQ37PPZ59FS22HGEZRZPY","nemo-evaluator-sdk",null,"Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.","math12345678\u002Fquanticle","math12345678","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F74202272?v=4","01M77XDK5Z6CNQM8T107PHDX1P","Smyan Reddy","01M77XDK6P25R6WF4RAGG08X9S",0,"MIT","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-evaluator-sdk\ndescription: Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Evaluation, NeMo, NVIDIA, Benchmarking, MMLU, HumanEval, Multi-Backend, Slurm, Docker, Reproducible, Enterprise]\ndependencies: [nemo-evaluator-launcher>=0.1.25, docker]\n---\n\n# NeMo Evaluator SDK - Enterprise LLM Benchmarking\n\n## Quick Start\n\nNeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).\n\n**Installation**:\n```bash\npip install nemo-evaluator-launcher\n```\n\n**Set API key and run evaluation**:\n```bash\nexport NGC_API_KEY=nvapi-your-key-here\n\n# Create minimal config\ncat > config.yaml \u003C\u003C 'EOF'\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n\nevaluation:\n  tasks:\n    - name: ifeval\nEOF\n\n# Run evaluation\nnemo-evaluator-launcher run --config-dir . --config-name config\n```\n\n**View available tasks**:\n```bash\nnemo-evaluator-launcher ls tasks\n```\n\n## Common Workflows\n\n### Workflow 1: Evaluate Model on Standard Benchmarks\n\nRun core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint.\n\n**Checklist**:\n```\nStandard Evaluation:\n- [ ] Step 1: Configure API endpoint\n- [ ] Step 2: Select benchmarks\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Check results\n```\n\n**Step 1: Configure API endpoint**\n\n```yaml\n# config.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n```\n\nFor self-hosted endpoints (vLLM, TRT-LLM):\n```yaml\ntarget:\n  api_endpoint:\n    model_id: my-model\n    url: http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: \"\"  # No key needed for local\n```\n\n**Step 2: Select benchmarks**\n\nAdd tasks to your config:\n```yaml\nevaluation:\n  tasks:\n    - name: ifeval           # Instruction following\n    - name: gpqa_diamond     # Graduate-level QA\n      env_vars:\n        HF_TOKEN: HF_TOKEN   # Some tasks need HF token\n    - name: gsm8k_cot_instruct  # Math reasoning\n    - name: humaneval        # Code generation\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Run with config file\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config\n\n# Override output directory\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o execution.output_dir=.\u002Fmy_results\n\n# Limit samples for quick testing\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=10\n```\n\n**Step 4: Check results**\n\n```bash\n# Check job status\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# List all runs\nnemo-evaluator-launcher ls runs\n\n# View results\ncat results\u002F\u003Cinvocation_id>\u002F\u003Ctask>\u002Fartifacts\u002Fresults.yml\n```\n\n### Workflow 2: Run Evaluation on Slurm HPC Cluster\n\nExecute large-scale evaluation on HPC infrastructure.\n\n**Checklist**:\n```\nSlurm Evaluation:\n- [ ] Step 1: Configure Slurm settings\n- [ ] Step 2: Set up model deployment\n- [ ] Step 3: Launch evaluation\n- [ ] Step 4: Monitor job status\n```\n\n**Step 1: Configure Slurm settings**\n\n```yaml\n# slurm_config.yaml\ndefaults:\n  - execution: slurm\n  - deployment: vllm\n  - _self_\n\nexecution:\n  hostname: cluster.example.com\n  account: my_slurm_account\n  partition: gpu\n  output_dir: \u002Fshared\u002Fresults\n  walltime: \"04:00:00\"\n  nodes: 1\n  gpus_per_node: 8\n```\n\n**Step 2: Set up model deployment**\n\n```yaml\ndeployment:\n  checkpoint_path: \u002Fshared\u002Fmodels\u002Fllama-3.1-8b\n  tensor_parallel_size: 2\n  data_parallel_size: 4\n  max_model_len: 4096\n\ntarget:\n  api_endpoint:\n    model_id: llama-3.1-8b\n    # URL auto-generated by deployment\n```\n\n**Step 3: Launch evaluation**\n\n```bash\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name slurm_config\n```\n\n**Step 4: Monitor job status**\n\n```bash\n# Check status (queries sacct)\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# View detailed info\nnemo-evaluator-launcher info \u003Cinvocation_id>\n\n# Kill if needed\nnemo-evaluator-launcher kill \u003Cinvocation_id>\n```\n\n### Workflow 3: Compare Multiple Models\n\nBenchmark multiple models on the same tasks for comparison.\n\n**Checklist**:\n```\nModel Comparison:\n- [ ] Step 1: Create base config\n- [ ] Step 2: Run evaluations with overrides\n- [ ] Step 3: Export and compare results\n```\n\n**Step 1: Create base config**\n\n```yaml\n# base_eval.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fcomparison_results\n\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.01\n        parallelism: 4\n  tasks:\n    - name: mmlu_pro\n    - name: gsm8k_cot_instruct\n    - name: ifeval\n```\n\n**Step 2: Run evaluations with model overrides**\n\n```bash\n# Evaluate Llama 3.1 8B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=meta\u002Fllama-3.1-8b-instruct \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\n# Evaluate Mistral 7B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=mistralai\u002Fmistral-7b-instruct-v0.3 \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n```\n\n**Step 3: Export and compare**\n\n```bash\n# Export to MLflow\nnemo-evaluator-launcher export \u003Cinvocation_id_1> --dest mlflow\nnemo-evaluator-launcher export \u003Cinvocation_id_2> --dest mlflow\n\n# Export to local JSON\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest local --format json\n\n# Export to Weights & Biases\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest wandb\n```\n\n### Workflow 4: Safety and Vision-Language Evaluation\n\nEvaluate models on safety benchmarks and VLM tasks.\n\n**Checklist**:\n```\nSafety\u002FVLM Evaluation:\n- [ ] Step 1: Configure safety tasks\n- [ ] Step 2: Set up VLM tasks (if applicable)\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Configure safety tasks**\n\n```yaml\nevaluation:\n  tasks:\n    - name: aegis              # Safety harness\n    - name: wildguard          # Safety classification\n    - name: garak              # Security probing\n```\n\n**Step 2: Configure VLM tasks**\n\n```yaml\n# For vision-language models\ntarget:\n  api_endpoint:\n    type: vlm  # Vision-language endpoint\n    model_id: nvidia\u002Fllama-3.2-90b-vision-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\nevaluation:\n  tasks:\n    - name: ocrbench           # OCR evaluation\n    - name: chartqa            # Chart understanding\n    - name: mmmu               # Multimodal understanding\n```\n\n## When to Use vs Alternatives\n\n**Use NeMo Evaluator when:**\n- Need **100+ benchmarks** from 18+ harnesses in one platform\n- Running evaluations on **Slurm HPC clusters** or cloud\n- Requiring **reproducible** containerized evaluation\n- Evaluating against **OpenAI-compatible APIs** (vLLM, TRT-LLM, NIMs)\n- Need **enterprise-grade** evaluation with result export (MLflow, W&B)\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: Simpler setup for quick local evaluation\n- **bigcode-evaluation-harness**: Focused only on code benchmarks\n- **HELM**: Stanford's broader evaluation (fairness, efficiency)\n- **Custom scripts**: Highly specialized domain evaluation\n\n## Supported Harnesses and Tasks\n\n| Harness | Task Count | Categories |\n|---------|-----------|------------|\n| `lm-evaluation-harness` | 60+ | MMLU, GSM8K, HellaSwag, ARC |\n| `simple-evals` | 20+ | GPQA, MATH, AIME |\n| `bigcode-evaluation-harness` | 25+ | HumanEval, MBPP, MultiPL-E |\n| `safety-harness` | 3 | Aegis, WildGuard |\n| `garak` | 1 | Security probing |\n| `vlmevalkit` | 6+ | OCRBench, ChartQA, MMMU |\n| `bfcl` | 6 | Function calling v2\u002Fv3 |\n| `mtbench` | 2 | Multi-turn conversation |\n| `livecodebench` | 10+ | Live coding evaluation |\n| `helm` | 15 | Medical domain |\n| `nemo-skills` | 8 | Math, science, agentic |\n\n## Common Issues\n\n**Issue: Container pull fails**\n\nEnsure NGC credentials are configured:\n```bash\ndocker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY\n```\n\n**Issue: Task requires environment variable**\n\nSome tasks need HF_TOKEN or JUDGE_API_KEY:\n```yaml\nevaluation:\n  tasks:\n    - name: gpqa_diamond\n      env_vars:\n        HF_TOKEN: HF_TOKEN  # Maps env var name to env var\n```\n\n**Issue: Evaluation timeout**\n\nIncrease parallelism or reduce samples:\n```bash\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100\n```\n\n**Issue: Slurm job not starting**\n\nCheck Slurm account and partition:\n```yaml\nexecution:\n  account: correct_account\n  partition: gpu\n  qos: normal  # May need specific QOS\n```\n\n**Issue: Different results than expected**\n\nVerify configuration matches reported settings:\n```yaml\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.0  # Deterministic\n        num_fewshot: 5    # Check paper's fewshot count\n```\n\n## CLI Reference\n\n| Command | Description |\n|---------|-------------|\n| `run` | Execute evaluation with config |\n| `status \u003Cid>` | Check job status |\n| `info \u003Cid>` | View detailed job info |\n| `ls tasks` | List available benchmarks |\n| `ls runs` | List all invocations |\n| `export \u003Cid>` | Export results (mlflow\u002Fwandb\u002Flocal) |\n| `kill \u003Cid>` | Terminate running job |\n\n## Configuration Override Examples\n\n```bash\n# Override model endpoint\n-o target.api_endpoint.model_id=my-model\n-o target.api_endpoint.url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n\n# Add evaluation parameters\n-o +evaluation.nemo_evaluator_config.config.params.temperature=0.5\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=50\n\n# Change execution settings\n-o execution.output_dir=\u002Fcustom\u002Fpath\n-o execution.mode=parallel\n\n# Dynamically set tasks\n-o 'evaluation.tasks=[{name: ifeval}, {name: gsm8k}]'\n```\n\n## Python API Usage\n\nFor programmatic evaluation without the CLI:\n\n```python\nfrom nemo_evaluator.core.evaluate import evaluate\nfrom nemo_evaluator.api.api_dataclasses import (\n    EvaluationConfig,\n    EvaluationTarget,\n    ApiEndpoint,\n    EndpointType,\n    ConfigParams\n)\n\n# Configure evaluation\neval_config = EvaluationConfig(\n    type=\"mmlu_pro\",\n    output_dir=\".\u002Fresults\",\n    params=ConfigParams(\n        limit_samples=10,\n        temperature=0.0,\n        max_new_tokens=1024,\n        parallelism=4\n    )\n)\n\n# Configure target endpoint\ntarget_config = EvaluationTarget(\n    api_endpoint=ApiEndpoint(\n        model_id=\"meta\u002Fllama-3.1-8b-instruct\",\n        url=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\",\n        type=EndpointType.CHAT,\n        api_key=\"nvapi-your-key-here\"\n    )\n)\n\n# Run evaluation\nresult = evaluate(eval_cfg=eval_config, target_cfg=target_config)\n```\n\n## Advanced Topics\n\n**Multi-backend execution**: See [references\u002Fexecution-backends.md](references\u002Fexecution-backends.md)\n**Configuration deep-dive**: See [references\u002Fconfiguration.md](references\u002Fconfiguration.md)\n**Adapter and interceptor system**: See [references\u002Fadapter-system.md](references\u002Fadapter-system.md)\n**Custom benchmark integration**: See [references\u002Fcustom-benchmarks.md](references\u002Fcustom-benchmarks.md)\n\n## Requirements\n\n- **Python**: 3.10-3.13\n- **Docker**: Required for local execution\n- **NGC API Key**: For pulling containers and using NVIDIA Build\n- **HF_TOKEN**: Required for some benchmarks (GPQA, MMLU)\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\n- **NGC Containers**: nvcr.io\u002Fnvidia\u002Feval-factory\u002F\n- **NVIDIA Build**: https:\u002F\u002Fbuild.nvidia.com (free hosted models)\n- **Documentation**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Ftree\u002Fmain\u002Fdocs\n","# NeMo Evaluator SDK - Enterprise LLM Benchmarking\n\n## Quick Start\n\nNeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).\n\n**Installation**:\n```bash\npip install nemo-evaluator-launcher\n```\n\n**Set API key and run evaluation**:\n```bash\nexport NGC_API_KEY=nvapi-your-key-here\n\n# Create minimal config\ncat > config.yaml \u003C\u003C 'EOF'\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n\nevaluation:\n  tasks:\n    - name: ifeval\nEOF\n\n# Run evaluation\nnemo-evaluator-launcher run --config-dir . --config-name config\n```\n\n**View available tasks**:\n```bash\nnemo-evaluator-launcher ls tasks\n```\n\n## Common Workflows\n\n### Workflow 1: Evaluate Model on Standard Benchmarks\n\nRun core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint.\n\n**Checklist**:\n```\nStandard Evaluation:\n- [ ] Step 1: Configure API endpoint\n- [ ] Step 2: Select benchmarks\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Check results\n```\n\n**Step 1: Configure API endpoint**\n\n```yaml\n# config.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n```\n\nFor self-hosted endpoints (vLLM, TRT-LLM):\n```yaml\ntarget:\n  api_endpoint:\n    model_id: my-model\n    url: http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: \"\"  # No key needed for local\n```\n\n**Step 2: Select benchmarks**\n\nAdd tasks to your config:\n```yaml\nevaluation:\n  tasks:\n    - name: ifeval           # Instruction following\n    - name: gpqa_diamond     # Graduate-level QA\n      env_vars:\n        HF_TOKEN: HF_TOKEN   # Some tasks need HF token\n    - name: gsm8k_cot_instruct  # Math reasoning\n    - name: humaneval        # Code generation\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Run with config file\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config\n\n# Override output directory\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o execution.output_dir=.\u002Fmy_results\n\n# Limit samples for quick testing\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=10\n```\n\n**Step 4: Check results**\n\n```bash\n# Check job status\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# List all runs\nnemo-evaluator-launcher ls runs\n\n# View results\ncat results\u002F\u003Cinvocation_id>\u002F\u003Ctask>\u002Fartifacts\u002Fresults.yml\n```\n\n### Workflow 2: Run Evaluation on Slurm HPC Cluster\n\nExecute large-scale evaluation on HPC infrastructure.\n\n**Checklist**:\n```\nSlurm Evaluation:\n- [ ] Step 1: Configure Slurm settings\n- [ ] Step 2: Set up model deployment\n- [ ] Step 3: Launch evaluation\n- [ ] Step 4: Monitor job status\n```\n\n**Step 1: Configure Slurm settings**\n\n```yaml\n# slurm_config.yaml\ndefaults:\n  - execution: slurm\n  - deployment: vllm\n  - _self_\n\nexecution:\n  hostname: cluster.example.com\n  account: my_slurm_account\n  partition: gpu\n  output_dir: \u002Fshared\u002Fresults\n  walltime: \"04:00:00\"\n  nodes: 1\n  gpus_per_node: 8\n```\n\n**Step 2: Set up model deployment**\n\n```yaml\ndeployment:\n  checkpoint_path: \u002Fshared\u002Fmodels\u002Fllama-3.1-8b\n  tensor_parallel_size: 2\n  data_parallel_size: 4\n  max_model_len: 4096\n\ntarget:\n  api_endpoint:\n    model_id: llama-3.1-8b\n    # URL auto-generated by deployment\n```\n\n**Step 3: Launch evaluation**\n\n```bash\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name slurm_config\n```\n\n**Step 4: Monitor job status**\n\n```bash\n# Check status (queries sacct)\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# View detailed info\nnemo-evaluator-launcher info \u003Cinvocation_id>\n\n# Kill if needed\nnemo-evaluator-launcher kill \u003Cinvocation_id>\n```\n\n### Workflow 3: Compare Multiple Models\n\nBenchmark multiple models on the same tasks for comparison.\n\n**Checklist**:\n```\nModel Comparison:\n- [ ] Step 1: Create base config\n- [ ] Step 2: Run evaluations with overrides\n- [ ] Step 3: Export and compare results\n```\n\n**Step 1: Create base config**\n\n```yaml\n# base_eval.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fcomparison_results\n\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.01\n        parallelism: 4\n  tasks:\n    - name: mmlu_pro\n    - name: gsm8k_cot_instruct\n    - name: ifeval\n```\n\n**Step 2: Run evaluations with model overrides**\n\n```bash\n# Evaluate Llama 3.1 8B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=meta\u002Fllama-3.1-8b-instruct \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\n# Evaluate Mistral 7B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=mistralai\u002Fmistral-7b-instruct-v0.3 \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n```\n\n**Step 3: Export and compare**\n\n```bash\n# Export to MLflow\nnemo-evaluator-launcher export \u003Cinvocation_id_1> --dest mlflow\nnemo-evaluator-launcher export \u003Cinvocation_id_2> --dest mlflow\n\n# Export to local JSON\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest local --format json\n\n# Export to Weights & Biases\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest wandb\n```\n\n### Workflow 4: Safety and Vision-Language Evaluation\n\nEvaluate models on safety benchmarks and VLM tasks.\n\n**Checklist**:\n```\nSafety\u002FVLM Evaluation:\n- [ ] Step 1: Configure safety tasks\n- [ ] Step 2: Set up VLM tasks (if applicable)\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Configure safety tasks**\n\n```yaml\nevaluation:\n  tasks:\n    - name: aegis              # Safety harness\n    - name: wildguard          # Safety classification\n    - name: garak              # Security probing\n```\n\n**Step 2: Configure VLM tasks**\n\n```yaml\n# For vision-language models\ntarget:\n  api_endpoint:\n    type: vlm  # Vision-language endpoint\n    model_id: nvidia\u002Fllama-3.2-90b-vision-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\nevaluation:\n  tasks:\n    - name: ocrbench           # OCR evaluation\n    - name: chartqa            # Chart understanding\n    - name: mmmu               # Multimodal understanding\n```\n\n## When to Use vs Alternatives\n\n**Use NeMo Evaluator when:**\n- Need **100+ benchmarks** from 18+ harnesses in one platform\n- Running evaluations on **Slurm HPC clusters** or cloud\n- Requiring **reproducible** containerized evaluation\n- Evaluating against **OpenAI-compatible APIs** (vLLM, TRT-LLM, NIMs)\n- Need **enterprise-grade** evaluation with result export (MLflow, W&B)\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: Simpler setup for quick local evaluation\n- **bigcode-evaluation-harness**: Focused only on code benchmarks\n- **HELM**: Stanford's broader evaluation (fairness, efficiency)\n- **Custom scripts**: Highly specialized domain evaluation\n\n## Supported Harnesses and Tasks\n\n| Harness | Task Count | Categories |\n|---------|-----------|------------|\n| `lm-evaluation-harness` | 60+ | MMLU, GSM8K, HellaSwag, ARC |\n| `simple-evals` | 20+ | GPQA, MATH, AIME |\n| `bigcode-evaluation-harness` | 25+ | HumanEval, MBPP, MultiPL-E |\n| `safety-harness` | 3 | Aegis, WildGuard |\n| `garak` | 1 | Security probing |\n| `vlmevalkit` | 6+ | OCRBench, ChartQA, MMMU |\n| `bfcl` | 6 | Function calling v2\u002Fv3 |\n| `mtbench` | 2 | Multi-turn conversation |\n| `livecodebench` | 10+ | Live coding evaluation |\n| `helm` | 15 | Medical domain |\n| `nemo-skills` | 8 | Math, science, agentic |\n\n## Common Issues\n\n**Issue: Container pull fails**\n\nEnsure NGC credentials are configured:\n```bash\ndocker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY\n```\n\n**Issue: Task requires environment variable**\n\nSome tasks need HF_TOKEN or JUDGE_API_KEY:\n```yaml\nevaluation:\n  tasks:\n    - name: gpqa_diamond\n      env_vars:\n        HF_TOKEN: HF_TOKEN  # Maps env var name to env var\n```\n\n**Issue: Evaluation timeout**\n\nIncrease parallelism or reduce samples:\n```bash\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100\n```\n\n**Issue: Slurm job not starting**\n\nCheck Slurm account and partition:\n```yaml\nexecution:\n  account: correct_account\n  partition: gpu\n  qos: normal  # May need specific QOS\n```\n\n**Issue: Different results than expected**\n\nVerify configuration matches reported settings:\n```yaml\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.0  # Deterministic\n        num_fewshot: 5    # Check paper's fewshot count\n```\n\n## CLI Reference\n\n| Command | Description |\n|---------|-------------|\n| `run` | Execute evaluation with config |\n| `status \u003Cid>` | Check job status |\n| `info \u003Cid>` | View detailed job info |\n| `ls tasks` | List available benchmarks |\n| `ls runs` | List all invocations |\n| `export \u003Cid>` | Export results (mlflow\u002Fwandb\u002Flocal) |\n| `kill \u003Cid>` | Terminate running job |\n\n## Configuration Override Examples\n\n```bash\n# Override model endpoint\n-o target.api_endpoint.model_id=my-model\n-o target.api_endpoint.url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n\n# Add evaluation parameters\n-o +evaluation.nemo_evaluator_config.config.params.temperature=0.5\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=50\n\n# Change execution settings\n-o execution.output_dir=\u002Fcustom\u002Fpath\n-o execution.mode=parallel\n\n# Dynamically set tasks\n-o 'evaluation.tasks=[{name: ifeval}, {name: gsm8k}]'\n```\n\n## Python API Usage\n\nFor programmatic evaluation without the CLI:\n\n```python\nfrom nemo_evaluator.core.evaluate import evaluate\nfrom nemo_evaluator.api.api_dataclasses import (\n    EvaluationConfig,\n    EvaluationTarget,\n    ApiEndpoint,\n    EndpointType,\n    ConfigParams\n)\n\n# Configure evaluation\neval_config = EvaluationConfig(\n    type=\"mmlu_pro\",\n    output_dir=\".\u002Fresults\",\n    params=ConfigParams(\n        limit_samples=10,\n        temperature=0.0,\n        max_new_tokens=1024,\n        parallelism=4\n    )\n)\n\n# Configure target endpoint\ntarget_config = EvaluationTarget(\n    api_endpoint=ApiEndpoint(\n        model_id=\"meta\u002Fllama-3.1-8b-instruct\",\n        url=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\",\n        type=EndpointType.CHAT,\n        api_key=\"nvapi-your-key-here\"\n    )\n)\n\n# Run evaluation\nresult = evaluate(eval_cfg=eval_config, target_cfg=target_config)\n```\n\n## Advanced Topics\n\n**Multi-backend execution**: See [references\u002Fexecution-backends.md](references\u002Fexecution-backends.md)\n**Configuration deep-dive**: See [references\u002Fconfiguration.md](references\u002Fconfiguration.md)\n**Adapter and interceptor system**: See [references\u002Fadapter-system.md](references\u002Fadapter-system.md)\n**Custom benchmark integration**: See [references\u002Fcustom-benchmarks.md](references\u002Fcustom-benchmarks.md)\n\n## Requirements\n\n- **Python**: 3.10-3.13\n- **Docker**: Required for local execution\n- **NGC API Key**: For pulling containers and using NVIDIA Build\n- **HF_TOKEN**: Required for some benchmarks (GPQA, MMLU)\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\n- **NGC Containers**: nvcr.io\u002Fnvidia\u002Feval-factory\u002F\n- **NVIDIA Build**: https:\u002F\u002Fbuild.nvidia.com (free hosted models)\n- **Documentation**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Ftree\u002Fmain\u002Fdocs\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Evaluation","NeMo","NVIDIA","Benchmarking","MMLU","HumanEval","Multi-Backend","Slurm","Docker","Reproducible","Enterprise","Orchestra Research","1.0.0",[39,40],"nemo-evaluator-launcher>=0.1.25","docker","main","3c55f1641d1423e5271b321e2fd40d3a5ed92fd2","quanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle","https:\u002F\u002Fgithub.com\u002Fmath12345678","I'm a 10th-grade student working at the intersection of computational biology, molecular virology, and AI-driven approaches to medicine and global health.","2026-06-30 15:02:45",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add math12345678\u002Fquanticle --skill nemo-evaluator-sdk",{"tool":54,"command":55},"bunx","bunx skills add math12345678\u002Fquanticle --skill nemo-evaluator-sdk",{"tool":57,"command":58},"pnpm","pnpm dlx skills add math12345678\u002Fquanticle --skill nemo-evaluator-sdk","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator\u002FSKILL.md",true,[62,68,72,76],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fadapter-system.md","REFERENCE","参考文档",9468,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator\u002Freferences\u002Fadapter-system.md",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fconfiguration.md",9567,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator\u002Freferences\u002Fconfiguration.md",{"path":73,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fcustom-benchmarks.md",7101,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator\u002Freferences\u002Fcustom-benchmarks.md",{"path":77,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fexecution-backends.md",9351,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F11-evaluation\u002Fnemo-evaluator\u002Freferences\u002Fexecution-backends.md",[81,88,95,102,109,116],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M77XYA93N9NZCDQQ58SCKQ49","ara-rigor-reviewer","Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong Accept-to-Reject recommendation. Use after Level 1 structural validation passes, when an ARA needs an objective epistemic critique before publication or release.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F22-agent-native-research-artifact\u002Frigor-reviewer\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M77XY5PY6DRG8YBZG24WQ039","ara-research-manager","Records research provenance as a post-task epilogue, scanning conversation history at the end of a coding or research session to extract decisions, experiments, dead ends, claims, heuristics, and pivots, and writing them into the ara\u002F directory with user-vs-AI provenance tags. Use as a session epilogue — never during execution — to maintain a faithful, auditable trace of how a research project actually evolved.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F22-agent-native-research-artifact\u002Fresearch-manager\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M77XY138ZMY0JG84569T8MSW","ara-compiler","Compiles any research input — PDF papers, GitHub repositories, experiment logs, code directories, or raw notes — into a complete Agent-Native Research Artifact (ARA) with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph, and grounded evidence. Use when ingesting a paper or codebase into a structured, machine-executable knowledge package, building an ARA from scratch, or converting research outputs into a falsifiable, agent-traversable form.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F22-agent-native-research-artifact\u002Fcompiler\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M77XXVQYZ3VVHM9KFFKB0PDB","creative-thinking-for-research","Applies cognitive science frameworks for creative thinking to CS and AI research ideation. Use when seeking genuinely novel research directions by leveraging combinatorial creativity, analogical reasoning, constraint manipulation, and other empirically grounded creative strategies.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F21-research-ideation\u002Fcreative-thinking-for-research\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M77XXPGJ9FR86MNBG74RKV3W","brainstorming-research-ideas","Guides researchers through structured ideation frameworks to discover high-impact research directions. Use when exploring new problem spaces, pivoting between projects, or seeking novel angles on existing work.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F21-research-ideation\u002Fbrainstorming-research-ideas\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M77XXG6R4E97DG3M1CT7CT0Q","systems-paper-writing","Comprehensive guide for writing systems papers targeting OSDI, SOSP, ASPLOS, NSDI, and EuroSys. Provides paragraph-level structural blueprints, writing patterns, venue-specific checklists, reviewer guidelines, LaTeX templates, and conference deadlines. Use this skill for all systems conference paper writing.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F20-ml-paper-writing\u002Fsystems-paper-writing\u002FSKILL.md",[],[]]