[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EDRSAX683GSVV8QK97ERFM":3,"skill-same:01M6EDRSAX683GSVV8QK97ERFM":48},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":21,"linkOnly":22,"markdownContent":23,"markdownBody":23,"frontMatter":24,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M6EDRSAX683GSVV8QK97ERFM","nemo-evaluator",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fnemo-evaluator\u002FSKILL.md",[],[18,19,20],"NeMo","evaluator","nemo evaluator","2026-11-10 10:12:30",false,"# NeMo Evaluator SDK - Enterprise LLM Benchmarking\n\n## Quick Start\n\nNeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).\n\n**Installation**:\n```bash\npip install nemo-evaluator-launcher\n```\n\n**Set API key and run evaluation**:\n```bash\nexport NGC_API_KEY=nvapi-your-key-here\n\n# Create minimal config\ncat > config.yaml \u003C\u003C 'EOF'\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n\nevaluation:\n  tasks:\n    - name: ifeval\nEOF\n\n# Run evaluation\nnemo-evaluator-launcher run --config-dir . --config-name config\n```\n\n**View available tasks**:\n```bash\nnemo-evaluator-launcher ls tasks\n```\n\n## Common Workflows\n\n### Workflow 1: Evaluate Model on Standard Benchmarks\n\nRun core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint.\n\n**Checklist**:\n```\nStandard Evaluation:\n- [ ] Step 1: Configure API endpoint\n- [ ] Step 2: Select benchmarks\n- [ ] Step 3: Run evaluation\n- [ ] Step 4: Check results\n```\n\n**Step 1: Configure API endpoint**\n\n```yaml\n# config.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fresults\n\ntarget:\n  api_endpoint:\n    model_id: meta\u002Fllama-3.1-8b-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: NGC_API_KEY\n```\n\nFor self-hosted endpoints (vLLM, TRT-LLM):\n```yaml\ntarget:\n  api_endpoint:\n    model_id: my-model\n    url: http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n    api_key_name: \"\"  # No key needed for local\n```\n\n**Step 2: Select benchmarks**\n\nAdd tasks to your config:\n```yaml\nevaluation:\n  tasks:\n    - name: ifeval           # Instruction following\n    - name: gpqa_diamond     # Graduate-level QA\n      env_vars:\n        HF_TOKEN: HF_TOKEN   # Some tasks need HF token\n    - name: gsm8k_cot_instruct  # Math reasoning\n    - name: humaneval        # Code generation\n```\n\n**Step 3: Run evaluation**\n\n```bash\n# Run with config file\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config\n\n# Override output directory\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o execution.output_dir=.\u002Fmy_results\n\n# Limit samples for quick testing\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name config \\\n  -o +evaluation.nemo_evaluator_config.config.params.limit_samples=10\n```\n\n**Step 4: Check results**\n\n```bash\n# Check job status\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# List all runs\nnemo-evaluator-launcher ls runs\n\n# View results\ncat results\u002F\u003Cinvocation_id>\u002F\u003Ctask>\u002Fartifacts\u002Fresults.yml\n```\n\n### Workflow 2: Run Evaluation on Slurm HPC Cluster\n\nExecute large-scale evaluation on HPC infrastructure.\n\n**Checklist**:\n```\nSlurm Evaluation:\n- [ ] Step 1: Configure Slurm settings\n- [ ] Step 2: Set up model deployment\n- [ ] Step 3: Launch evaluation\n- [ ] Step 4: Monitor job status\n```\n\n**Step 1: Configure Slurm settings**\n\n```yaml\n# slurm_config.yaml\ndefaults:\n  - execution: slurm\n  - deployment: vllm\n  - _self_\n\nexecution:\n  hostname: cluster.example.com\n  account: my_slurm_account\n  partition: gpu\n  output_dir: \u002Fshared\u002Fresults\n  walltime: \"04:00:00\"\n  nodes: 1\n  gpus_per_node: 8\n```\n\n**Step 2: Set up model deployment**\n\n```yaml\ndeployment:\n  checkpoint_path: \u002Fshared\u002Fmodels\u002Fllama-3.1-8b\n  tensor_parallel_size: 2\n  data_parallel_size: 4\n  max_model_len: 4096\n\ntarget:\n  api_endpoint:\n    model_id: llama-3.1-8b\n    # URL auto-generated by deployment\n```\n\n**Step 3: Launch evaluation**\n\n```bash\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name slurm_config\n```\n\n**Step 4: Monitor job status**\n\n```bash\n# Check status (queries sacct)\nnemo-evaluator-launcher status \u003Cinvocation_id>\n\n# View detailed info\nnemo-evaluator-launcher info \u003Cinvocation_id>\n\n# Kill if needed\nnemo-evaluator-launcher kill \u003Cinvocation_id>\n```\n\n### Workflow 3: Compare Multiple Models\n\nBenchmark multiple models on the same tasks for comparison.\n\n**Checklist**:\n```\nModel Comparison:\n- [ ] Step 1: Create base config\n- [ ] Step 2: Run evaluations with overrides\n- [ ] Step 3: Export and compare results\n```\n\n**Step 1: Create base config**\n\n```yaml\n# base_eval.yaml\ndefaults:\n  - execution: local\n  - deployment: none\n  - _self_\n\nexecution:\n  output_dir: .\u002Fcomparison_results\n\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.01\n        parallelism: 4\n  tasks:\n    - name: mmlu_pro\n    - name: gsm8k_cot_instruct\n    - name: ifeval\n```\n\n**Step 2: Run evaluations with model overrides**\n\n```bash\n# Evaluate Llama 3.1 8B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=meta\u002Fllama-3.1-8b-instruct \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\n# Evaluate Mistral 7B\nnemo-evaluator-launcher run \\\n  --config-dir . \\\n  --config-name base_eval \\\n  -o target.api_endpoint.model_id=mistralai\u002Fmistral-7b-instruct-v0.3 \\\n  -o target.api_endpoint.url=https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n```\n\n**Step 3: Export and compare**\n\n```bash\n# Export to MLflow\nnemo-evaluator-launcher export \u003Cinvocation_id_1> --dest mlflow\nnemo-evaluator-launcher export \u003Cinvocation_id_2> --dest mlflow\n\n# Export to local JSON\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest local --format json\n\n# Export to Weights & Biases\nnemo-evaluator-launcher export \u003Cinvocation_id> --dest wandb\n```\n\n### Workflow 4: Safety and Vision-Language Evaluation\n\nEvaluate models on safety benchmarks and VLM tasks.\n\n**Checklist**:\n```\nSafety\u002FVLM Evaluation:\n- [ ] Step 1: Configure safety tasks\n- [ ] Step 2: Set up VLM tasks (if applicable)\n- [ ] Step 3: Run evaluation\n```\n\n**Step 1: Configure safety tasks**\n\n```yaml\nevaluation:\n  tasks:\n    - name: aegis              # Safety harness\n    - name: wildguard          # Safety classification\n    - name: garak              # Security probing\n```\n\n**Step 2: Configure VLM tasks**\n\n```yaml\n# For vision-language models\ntarget:\n  api_endpoint:\n    type: vlm  # Vision-language endpoint\n    model_id: nvidia\u002Fllama-3.2-90b-vision-instruct\n    url: https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\n\nevaluation:\n  tasks:\n    - name: ocrbench           # OCR evaluation\n    - name: chartqa            # Chart understanding\n    - name: mmmu               # Multimodal understanding\n```\n\n## When to Use vs Alternatives\n\n**Use NeMo Evaluator when:**\n- Need **100+ benchmarks** from 18+ harnesses in one platform\n- Running evaluations on **Slurm HPC clusters** or cloud\n- Requiring **reproducible** containerized evaluation\n- Evaluating against **OpenAI-compatible APIs** (vLLM, TRT-LLM, NIMs)\n- Need **enterprise-grade** evaluation with result export (MLflow, W&B)\n\n**Use alternatives instead:**\n- **lm-evaluation-harness**: Simpler setup for quick local evaluation\n- **bigcode-evaluation-harness**: Focused only on code benchmarks\n- **HELM**: Stanford's broader evaluation (fairness, efficiency)\n- **Custom scripts**: Highly specialized domain evaluation\n\n## Supported Harnesses and Tasks\n\n| Harness | Task Count | Categories |\n|---------|-----------|------------|\n| `lm-evaluation-harness` | 60+ | MMLU, GSM8K, HellaSwag, ARC |\n| `simple-evals` | 20+ | GPQA, MATH, AIME |\n| `bigcode-evaluation-harness` | 25+ | HumanEval, MBPP, MultiPL-E |\n| `safety-harness` | 3 | Aegis, WildGuard |\n| `garak` | 1 | Security probing |\n| `vlmevalkit` | 6+ | OCRBench, ChartQA, MMMU |\n| `bfcl` | 6 | Function calling v2\u002Fv3 |\n| `mtbench` | 2 | Multi-turn conversation |\n| `livecodebench` | 10+ | Live coding evaluation |\n| `helm` | 15 | Medical domain |\n| `nemo-skills` | 8 | Math, science, agentic |\n\n## Common Issues\n\n**Issue: Container pull fails**\n\nEnsure NGC credentials are configured:\n```bash\ndocker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY\n```\n\n**Issue: Task requires environment variable**\n\nSome tasks need HF_TOKEN or JUDGE_API_KEY:\n```yaml\nevaluation:\n  tasks:\n    - name: gpqa_diamond\n      env_vars:\n        HF_TOKEN: HF_TOKEN  # Maps env var name to env var\n```\n\n**Issue: Evaluation timeout**\n\nIncrease parallelism or reduce samples:\n```bash\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100\n```\n\n**Issue: Slurm job not starting**\n\nCheck Slurm account and partition:\n```yaml\nexecution:\n  account: correct_account\n  partition: gpu\n  qos: normal  # May need specific QOS\n```\n\n**Issue: Different results than expected**\n\nVerify configuration matches reported settings:\n```yaml\nevaluation:\n  nemo_evaluator_config:\n    config:\n      params:\n        temperature: 0.0  # Deterministic\n        num_fewshot: 5    # Check paper's fewshot count\n```\n\n## CLI Reference\n\n| Command | Description |\n|---------|-------------|\n| `run` | Execute evaluation with config |\n| `status \u003Cid>` | Check job status |\n| `info \u003Cid>` | View detailed job info |\n| `ls tasks` | List available benchmarks |\n| `ls runs` | List all invocations |\n| `export \u003Cid>` | Export results (mlflow\u002Fwandb\u002Flocal) |\n| `kill \u003Cid>` | Terminate running job |\n\n## Configuration Override Examples\n\n```bash\n# Override model endpoint\n-o target.api_endpoint.model_id=my-model\n-o target.api_endpoint.url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions\n\n# Add evaluation parameters\n-o +evaluation.nemo_evaluator_config.config.params.temperature=0.5\n-o +evaluation.nemo_evaluator_config.config.params.parallelism=8\n-o +evaluation.nemo_evaluator_config.config.params.limit_samples=50\n\n# Change execution settings\n-o execution.output_dir=\u002Fcustom\u002Fpath\n-o execution.mode=parallel\n\n# Dynamically set tasks\n-o 'evaluation.tasks=[{name: ifeval}, {name: gsm8k}]'\n```\n\n## Python API Usage\n\nFor programmatic evaluation without the CLI:\n\n```python\nfrom nemo_evaluator.core.evaluate import evaluate\nfrom nemo_evaluator.api.api_dataclasses import (\n    EvaluationConfig,\n    EvaluationTarget,\n    ApiEndpoint,\n    EndpointType,\n    ConfigParams\n)\n\n# Configure evaluation\neval_config = EvaluationConfig(\n    type=\"mmlu_pro\",\n    output_dir=\".\u002Fresults\",\n    params=ConfigParams(\n        limit_samples=10,\n        temperature=0.0,\n        max_new_tokens=1024,\n        parallelism=4\n    )\n)\n\n# Configure target endpoint\ntarget_config = EvaluationTarget(\n    api_endpoint=ApiEndpoint(\n        model_id=\"meta\u002Fllama-3.1-8b-instruct\",\n        url=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\u002Fchat\u002Fcompletions\",\n        type=EndpointType.CHAT,\n        api_key=\"nvapi-your-key-here\"\n    )\n)\n\n# Run evaluation\nresult = evaluate(eval_cfg=eval_config, target_cfg=target_config)\n```\n\n## Advanced Topics\n\n**Multi-backend execution**: See [references\u002Fexecution-backends.md](references\u002Fexecution-backends.md)\n**Configuration deep-dive**: See [references\u002Fconfiguration.md](references\u002Fconfiguration.md)\n**Adapter and interceptor system**: See [references\u002Fadapter-system.md](references\u002Fadapter-system.md)\n**Custom benchmark integration**: See [references\u002Fcustom-benchmarks.md](references\u002Fcustom-benchmarks.md)\n\n## Requirements\n\n- **Python**: 3.10-3.13\n- **Docker**: Required for local execution\n- **NGC API Key**: For pulling containers and using NVIDIA Build\n- **HF_TOKEN**: Required for some benchmarks (GPQA, MMLU)\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\n- **NGC Containers**: nvcr.io\u002Fnvidia\u002Feval-factory\u002F\n- **NVIDIA Build**: https:\u002F\u002Fbuild.nvidia.com (free hosted models)\n- **Documentation**: https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Ftree\u002Fmain\u002Fdocs\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fnemo-evaluator",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill nemo-evaluator",{"tool":40,"command":41},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill nemo-evaluator",{"tool":43,"command":44},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill nemo-evaluator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fnemo-evaluator\u002FSKILL.md",true,[],[49,63,73,82,88,95],{"skillNo":50,"name":51,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":59,"publishedAt":21},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[54],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[60,61,62],"model","model hierarchy","model hierarchy skill",{"skillNo":64,"name":65,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":69,"publishedAt":21},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[68],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},[70,71,72],"MCP","zoom","zoom mcp",{"skillNo":74,"name":75,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":21},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[79,80,81,71],"sdk","zoom apps sdk","apps",{"skillNo":83,"name":84,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":21},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[84],{"skillNo":89,"name":90,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":94,"publishedAt":21},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[93],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},[90],{"skillNo":96,"name":97,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":21},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[97]]