[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CPSFTH7B3MFGWR5KN67BJS":3,"skill-same:01M7CPSFTH7B3MFGWR5KN67BJS":49},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":6,"ownerDescription":33,"updatedAt":34,"install":35,"resources":48},"01M7CPSFTH7B3MFGWR5KN67BJS","nemo-evaluator-plugin",null,"Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.","om-scogo\u002Fskillsh-scraper","om-scogo","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160466840?v=4","01M79AWB6DFVCQ7RK70CS34B96","Om Achrekar","01M79AWB6ZM9SKR8KDH1WSSM22",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fnvidia-skills\u002Fnemo-evaluator-plugin\u002Fskills\u002Fnemo-evaluator-plugin\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-evaluator-plugin\ndescription: Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.\nmetadata:\n  owner: nemo-platform\n  maturity: active\nlicense: Apache-2.0\n---\n\n# Evaluator Plugin\n\nUse this skill for evaluation tasks against a running NeMo Platform server. The plugin-backed CLI interface is `nemo evaluator`; the legacy generated `nemo evaluation` API command group is not the target surface for new guidance.\n\n## CLI Interface\n\n### Prerequisites\n\n- all commands in this file assume that the shell's working dir is at the root of the Nvidia-NeMo\u002Fnemo-platform repo\n- activate the Python virtual environment before invoking the `nemo` CLI: `source .venv\u002Fbin\u002Factivate`\n\nCheck plugin status from the CLI:\n\n```bash\nnemo evaluator info\n```\n\n## Metric Types\n\n### Explore Available Metrics\n\nTo view available metric names, run:\n\n```bash\nnemo evaluator metric-types\n```\n\nTo view a specific metric schema, pass a metric name from the `metric_types` list above:\n\n```bash\nnemo evaluator metric-types \u003Cmetric-name>\n```\n\nInspect all the registered metric schema contracts:\n\n```bash\nnemo evaluator evaluate explain\n```\n\n> Note: use `nemo evaluator evaluate explain` as the source of truth for the current plugin input schema. It will return a large json schema response, so strongly prefer `nemo evaluator metric-types` when you only need metric names and corresponding schemas.\n\n## Evaluation Spec\n\nEvaluation spec is a payload that is provided to CLI as an input to execute evaluation.\n\nAt a high level, a spec describes:\n\n- `metrics`: bundled Evaluator SDK metric configurations\n- `dataset`: inline rows to evaluate or platform FilesetRef that contains the dataset\n- `params`: optional Evaluator SDK execution parameters\n- `target`: optional model or agent target for online evaluation\n\nSee the LLM-judge spec example at [assets\u002Fspecs\u002Fllm_as_judge.json](.\u002Fassets\u002Fspecs\u002Fllm_as_judge.json).\n\n### Metric Bundle Payloads\n\nThe checked-in [spec examples](.\u002Fassets\u002Fspecs) use bundled SDK metrics. The fields under `metrics[*].payload` are generated by `bundle_metric(metric, CloudpickleMetricBundlePackager())`.\n\nTo see the pattern for configuring a pre-defined SDK metric, for example `ExactMatchMetric`, and converting it into bundled metric JSON, inspect `build_metric_bundle_example()` in [generate_example_specs.py](.\u002Fscripts\u002Fgenerate_example_specs.py) and run:\n\n```bash\nuv run --frozen python skills\u002Fnemo-evaluator-plugin\u002Fscripts\u002Fgenerate_example_specs.py\n```\n\n## Run Evaluations\n\n### Run Using File Spec Reference\n\nWhen using the `nemo evaluator evaluate run` command, results are saved into local temporary directories and the link is printed to stdout.\nPrefer the `--spec-file` named argument over inline shell JSON because metric bundles include serialized payloads.\nExamples of various specs are provided in the [assets\u002Fspecs](.\u002Fassets\u002Fspecs\u002F) directory.\n\n#### Evaluate using `exact-match` metric\n\nSee the spec example at [assets\u002Fspecs\u002Fexact_match_metric.json](.\u002Fassets\u002Fspecs\u002Fexact_match_metric.json).\n\n```bash\nnemo evaluator evaluate run --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fexact_match_metric.json\n```\n\n#### Evaluate using a benchmark metric set\n\n```bash\nnemo evaluator evaluate run --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fexact_match_benchmark.json\n```\n\n#### Evaluate using `LLM-Judge` metric\n\nUses an LLM to score responses. See the spec example at [assets\u002Fspecs\u002Fllm_as_judge.json](.\u002Fassets\u002Fspecs\u002Fllm_as_judge.json).\n\n```bash\nnemo evaluator evaluate run --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fllm_as_judge.json\n```\n\n### Run Evaluation As A Durable Job\n\nUse the `nemo evaluator evaluate submit` command to create a durable evaluation job. The response of this command returns a job handler object instead of the evaluation result.\n\n```bash\nnemo evaluator evaluate submit \\\n  --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fexact_match_metric.json\n```\n\nThe submit response includes the generated job's `name` field, for example `nemo-evaluator-zlhn1ecd`. Wait for the job to complete, then list and download the job results.\n\n```bash\nnemo jobs get-status \u003Cjob-name>\nnemo jobs get \u003Cjob-name>\nnemo jobs results list \u003Cjob-name>\nnemo jobs results download aggregate-scores --job \u003Cjob-name> --output-file aggregate-scores.json\nnemo jobs results download row-scores --job \u003Cjob-name> --output-file row-scores.jsonl\n```\n\n## Python SDK Interface\n\nEvaluator Python SDK client is exposed as `evaluator` variable on `NeMoPlatform` instance:\n\n```python\nfrom nemo_platform import NeMoPlatform\n\nplatform_client = NeMoPlatform(base_url=\"http:\u002F\u002Flocalhost:8080\")\nstatus = platform_client.evaluator.plugin_status()\n```\n\nSee examples of using the plugin SDK interface in [plugin_sdk_examples.py](.\u002Fassets\u002Fexamples\u002Fplugin_sdk_examples.py).\n\n## Security\nMake sure not to print any secrets to stdout since this can be collected as logs\n\n## Additional Resources\n\nFor LLM-judge setup notes, see [LLM Judge Notes](references\u002Fllm-judge.md).\n\nFor evaluator API key auth, see [Evaluator API Auth](references\u002Fapi-auth.md).\n\nFor local and cluster troubleshooting, see [Evaluation Troubleshooting](references\u002Ftroubleshooting.md).\n","# Evaluator Plugin\n\nUse this skill for evaluation tasks against a running NeMo Platform server. The plugin-backed CLI interface is `nemo evaluator`; the legacy generated `nemo evaluation` API command group is not the target surface for new guidance.\n\n## CLI Interface\n\n### Prerequisites\n\n- all commands in this file assume that the shell's working dir is at the root of the Nvidia-NeMo\u002Fnemo-platform repo\n- activate the Python virtual environment before invoking the `nemo` CLI: `source .venv\u002Fbin\u002Factivate`\n\nCheck plugin status from the CLI:\n\n```bash\nnemo evaluator info\n```\n\n## Metric Types\n\n### Explore Available Metrics\n\nTo view available metric names, run:\n\n```bash\nnemo evaluator metric-types\n```\n\nTo view a specific metric schema, pass a metric name from the `metric_types` list above:\n\n```bash\nnemo evaluator metric-types \u003Cmetric-name>\n```\n\nInspect all the registered metric schema contracts:\n\n```bash\nnemo evaluator evaluate explain\n```\n\n> Note: use `nemo evaluator evaluate explain` as the source of truth for the current plugin input schema. It will return a large json schema response, so strongly prefer `nemo evaluator metric-types` when you only need metric names and corresponding schemas.\n\n## Evaluation Spec\n\nEvaluation spec is a payload that is provided to CLI as an input to execute evaluation.\n\nAt a high level, a spec describes:\n\n- `metrics`: bundled Evaluator SDK metric configurations\n- `dataset`: inline rows to evaluate or platform FilesetRef that contains the dataset\n- `params`: optional Evaluator SDK execution parameters\n- `target`: optional model or agent target for online evaluation\n\nSee the LLM-judge spec example at [assets\u002Fspecs\u002Fllm_as_judge.json](.\u002Fassets\u002Fspecs\u002Fllm_as_judge.json).\n\n### Metric Bundle Payloads\n\nThe checked-in [spec examples](.\u002Fassets\u002Fspecs) use bundled SDK metrics. The fields under `metrics[*].payload` are generated by `bundle_metric(metric, CloudpickleMetricBundlePackager())`.\n\nTo see the pattern for configuring a pre-defined SDK metric, for example `ExactMatchMetric`, and converting it into bundled metric JSON, inspect `build_metric_bundle_example()` in [generate_example_specs.py](.\u002Fscripts\u002Fgenerate_example_specs.py) and run:\n\n```bash\nuv run --frozen python skills\u002Fnemo-evaluator-plugin\u002Fscripts\u002Fgenerate_example_specs.py\n```\n\n## Run Evaluations\n\n### Run Using File Spec Reference\n\nWhen using the `nemo evaluator evaluate run` command, results are saved into local temporary directories and the link is printed to stdout.\nPrefer the `--spec-file` named argument over inline shell JSON because metric bundles include serialized payloads.\nExamples of various specs are provided in the [assets\u002Fspecs](.\u002Fassets\u002Fspecs\u002F) directory.\n\n#### Evaluate using `exact-match` metric\n\nSee the spec example at [assets\u002Fspecs\u002Fexact_match_metric.json](.\u002Fassets\u002Fspecs\u002Fexact_match_metric.json).\n\n```bash\nnemo evaluator evaluate run --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fexact_match_metric.json\n```\n\n#### Evaluate using a benchmark metric set\n\n```bash\nnemo evaluator evaluate run --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fexact_match_benchmark.json\n```\n\n#### Evaluate using `LLM-Judge` metric\n\nUses an LLM to score responses. See the spec example at [assets\u002Fspecs\u002Fllm_as_judge.json](.\u002Fassets\u002Fspecs\u002Fllm_as_judge.json).\n\n```bash\nnemo evaluator evaluate run --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fllm_as_judge.json\n```\n\n### Run Evaluation As A Durable Job\n\nUse the `nemo evaluator evaluate submit` command to create a durable evaluation job. The response of this command returns a job handler object instead of the evaluation result.\n\n```bash\nnemo evaluator evaluate submit \\\n  --spec-file skills\u002Fnemo-evaluator-plugin\u002Fassets\u002Fspecs\u002Fexact_match_metric.json\n```\n\nThe submit response includes the generated job's `name` field, for example `nemo-evaluator-zlhn1ecd`. Wait for the job to complete, then list and download the job results.\n\n```bash\nnemo jobs get-status \u003Cjob-name>\nnemo jobs get \u003Cjob-name>\nnemo jobs results list \u003Cjob-name>\nnemo jobs results download aggregate-scores --job \u003Cjob-name> --output-file aggregate-scores.json\nnemo jobs results download row-scores --job \u003Cjob-name> --output-file row-scores.jsonl\n```\n\n## Python SDK Interface\n\nEvaluator Python SDK client is exposed as `evaluator` variable on `NeMoPlatform` instance:\n\n```python\nfrom nemo_platform import NeMoPlatform\n\nplatform_client = NeMoPlatform(base_url=\"http:\u002F\u002Flocalhost:8080\")\nstatus = platform_client.evaluator.plugin_status()\n```\n\nSee examples of using the plugin SDK interface in [plugin_sdk_examples.py](.\u002Fassets\u002Fexamples\u002Fplugin_sdk_examples.py).\n\n## Security\nMake sure not to print any secrets to stdout since this can be collected as logs\n\n## Additional Resources\n\nFor LLM-judge setup notes, see [LLM Judge Notes](references\u002Fllm-judge.md).\n\nFor evaluator API key auth, see [Evaluator API Auth](references\u002Fapi-auth.md).\n\nFor local and cluster troubleshooting, see [Evaluation Troubleshooting](references\u002Ftroubleshooting.md).\n",{"name":5,"license":15,"metadata":24,"description":7},{"owner":25,"maturity":26},"nemo-platform","active","main","8859218955d9504cd18b59e7892100718039d3b4","data\u002Fskills\u002Fnvidia-skills\u002Fnemo-evaluator-plugin\u002Fskills\u002Fnemo-evaluator-plugin",1,"https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper","https:\u002F\u002Fgithub.com\u002Fom-scogo","SDE-1 @ Scogo ","2026-07-18 12:22:46",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add om-scogo\u002Fskillsh-scraper --skill nemo-evaluator-plugin",{"tool":41,"command":42},"bunx","bunx skills add om-scogo\u002Fskillsh-scraper --skill nemo-evaluator-plugin",{"tool":44,"command":45},"pnpm","pnpm dlx skills add om-scogo\u002Fskillsh-scraper --skill nemo-evaluator-plugin","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fnvidia-skills\u002Fnemo-evaluator-plugin\u002Fskills\u002Fnemo-evaluator-plugin\u002FSKILL.md",true,[],[50,58,63,70,75,82],{"skillNo":51,"name":52,"nameZh":6,"description":53,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":54,"licenseFileUrl":6,"sourceHtmlUrl":55,"categories":56,"keywords":57,"publishedAt":19},"01M7D6MC9FAAEV4CMNYX8W7JTG","arxiv-search","Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.","MIT","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fyorkeccak-scientific-skills\u002Farxiv-search\u002Fskills\u002Farxiv-search\u002FSKILL.md",[],[],{"skillNo":59,"name":52,"nameZh":6,"description":53,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":54,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7D6M85CKKN87TA98VCQV4H7","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fyorkeccak-scientific-skills\u002Farxiv-search\u002FSkill.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":54,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7D4ZTCS4DR3CE4JY06X84T2","working-with-legacy-code","Safely change and test untested codebases using Feathers' \"Working Effectively with Legacy Code\". Use when the user mentions \"legacy code\", \"no tests\", \"untested codebase\", \"how do I test this\", \"seams\", \"characterization tests\", \"golden master\", \"sprout method\", \"afraid to change this code\", \"monster method\", \"dependency breaking\", or \"inherited a messy codebase\". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout\u002Fwrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002Fworking-with-legacy-code\u002FSKILL.md",[],[],{"skillNo":71,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":54,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":19},"01M7D4ZHZKJD2RSNGAF63V14P4","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002FSkill.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":54,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M7D4ZB4CJBFW5JE67CPH2QRV","web-typography","Select, pair, and implement typefaces for web projects. Use when the user mentions \"font pairing\", \"which typeface\", \"line height\", \"responsive typography\", \"web font loading\", \"type hierarchy\", \"variable fonts\", \"FOUT\u002FFOIT\", \"typographic scale\", or \"the text is hard to read\". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002Fweb-typography\u002FSKILL.md",[],[],{"skillNo":83,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":54,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7D4Z4MVYVY3NMJ18BNV6WNA","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002FSkill.md",[],[]]