nemo-evaluator-plugin
Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- nemo-evaluator-plugin
- license
- Apache-2.0
- metadata
- { "owner": "nemo-platform", "maturity": "active" }
- description
- Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.
02 资源文件 12
- 脚本assets/examples/plugin_sdk_examples.py3.8 KB
- 资产文件assets/specs/exact_match_benchmark.json4.3 KB
- 资产文件assets/specs/exact_match_metric.json1.8 KB
- 资产文件assets/specs/llm_as_judge.json4.9 KB
- 参考文档BENCHMARK.md4.1 KB
- 参考文档metadata.json330 B
- 参考文档references/api-auth.md895 B
- 参考文档references/llm-judge.md1.2 KB
- 参考文档references/troubleshooting.md1.5 KB
- 脚本scripts/generate_example_specs.py1.6 KB
- 参考文档skill-card.md3.9 KB
- 参考文档verification.json2.6 KB
03 SKILL.md 正文
Evaluator Plugin
Use this skill for evaluation tasks against a running NeMo Platform server. The plugin-backed CLI interface is nemo evaluator; the legacy generated nemo evaluation API command group is not the target surface for new guidance.
CLI Interface
Prerequisites
- all commands in this file assume that the shell's working dir is at the root of the Nvidia-NeMo/nemo-platform repo
- activate the Python virtual environment before invoking the
nemoCLI:source .venv/bin/activate
Check plugin status from the CLI:
nemo evaluator info
Metric Types
Explore Available Metrics
To view available metric names, run:
nemo evaluator metric-types
To view a specific metric schema, pass a metric name from the metric_types list above:
nemo evaluator metric-types <metric-name>
Inspect all the registered metric schema contracts:
nemo evaluator evaluate explain
Note: use
nemo evaluator evaluate explainas the source of truth for the current plugin input schema. It will return a large json schema response, so strongly prefernemo evaluator metric-typeswhen you only need metric names and corresponding schemas.
Evaluation Spec
Evaluation spec is a payload that is provided to CLI as an input to execute evaluation.
At a high level, a spec describes:
metrics: bundled Evaluator SDK metric configurationsdataset: inline rows to evaluate or platform FilesetRef that contains the datasetparams: optional Evaluator SDK execution parameterstarget: optional model or agent target for online evaluation
See the LLM-judge spec example at assets/specs/llm_as_judge.json.
Metric Bundle Payloads
The checked-in spec examples use bundled SDK metrics. The fields under metrics[*].payload are generated by bundle_metric(metric, CloudpickleMetricBundlePackager()).
To see the pattern for configuring a pre-defined SDK metric, for example ExactMatchMetric, and converting it into bundled metric JSON, inspect build_metric_bundle_example() in generate_example_specs.py and run:
uv run --frozen python skills/nemo-evaluator-plugin/scripts/generate_example_specs.py
Run Evaluations
Run Using File Spec Reference
When using the nemo evaluator evaluate run command, results are saved into local temporary directories and the link is printed to stdout.
Prefer the --spec-file named argument over inline shell JSON because metric bundles include serialized payloads.
Examples of various specs are provided in the assets/specs directory.
Evaluate using exact-match metric
See the spec example at assets/specs/exact_match_metric.json.
nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json
Evaluate using a benchmark metric set
nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.json
Evaluate using LLM-Judge metric
Uses an LLM to score responses. See the spec example at assets/specs/llm_as_judge.json.
nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/llm_as_judge.json
Run Evaluation As A Durable Job
Use the nemo evaluator evaluate submit command to create a durable evaluation job. The response of this command returns a job handler object instead of the evaluation result.
nemo evaluator evaluate submit \
--spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json
The submit response includes the generated job's name field, for example nemo-evaluator-zlhn1ecd. Wait for the job to complete, then list and download the job results.
nemo jobs get-status <job-name>
nemo jobs get <job-name>
nemo jobs results list <job-name>
nemo jobs results download aggregate-scores --job <job-name> --output-file aggregate-scores.json
nemo jobs results download row-scores --job <job-name> --output-file row-scores.jsonl
Python SDK Interface
Evaluator Python SDK client is exposed as evaluator variable on NeMoPlatform instance:
from nemo_platform import NeMoPlatform
platform_client = NeMoPlatform(base_url="http://localhost:8080")
status = platform_client.evaluator.plugin_status()
See examples of using the plugin SDK interface in plugin_sdk_examples.py.
Security
Make sure not to print any secrets to stdout since this can be collected as logs
Additional Resources
For LLM-judge setup notes, see LLM Judge Notes.
For evaluator API key auth, see Evaluator API Auth.
For local and cluster troubleshooting, see Evaluation Troubleshooting.
04 同仓库更多 Skills
testing
For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.
qedgen
Formally verify programs by writing Lean 4 proofs. Trigger this skill whenever the user wants to formally verify code, generate Lean 4 proofs, prove properties about algorithms or smart contracts, verify invariants, convert program logic into formal specifications, or anything involving Lean 4 and formal verification. Also trigger when the user mentions "qedgen", "lean proof", "formal proof", "verify my code", "prove correctness", "formal verification", or wants mathematical guarantees about their implementation.
solana-dev
Use when user asks to "build a Solana dapp", "write an Anchor program", "create a token", "debug Solana errors", "set up wallet connection", "test my Solana program", "deploy to devnet", or "explain Solana concepts" (rent, accounts, PDAs, CPIs, etc.). Also use for quick on-chain lookups via public RPC + curl — "what's the balance of <wallet>", "look up transaction <sig>", "token balance for <account>", "check this address on mainnet/devnet". End-to-end Solana development playbook covering wallet connection, Anchor/Pinocchio programs, Codama client generation, LiteSVM/Mollusk/Surfpool testing, security checklists, and JSON-RPC curl lookups against public clusters. Integrates with the Solana MCP server for live documentation search. Prefers framework-kit (@solana/client + @solana/react-hooks) for UI, wallet-standard-first connection (incl. ConnectorKit), @solana/kit for client/RPC code, and @solana/web3-compat for legacy boundaries.
solana-common-errors
Diagnose and fix common Solana development errors — GLIBC mismatches, Anchor version conflicts, cargo build-sbf / platform-tools failures, LiteSVM issues, edition2024 crate pins, RPC airdrops, and Anchor 0.29→0.32 migrations. Use when builds fail, Anchor CLI won't install, or CI breaks on Debian/Ubuntu.
skill-creator
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
phantom-wallet-mcp
Execute wallet operations through the Phantom MCP server. Use when the user wants to interact with their Phantom wallet directly — get addresses, transfer SOL or SPL tokens, buy/swap tokens, sign transactions, and sign messages across Solana, Ethereum, Bitcoin, and Sui. Requires the @phantom/mcp-server to be configured as an MCP server.