AGENT SKILL

nemo-evaluator-plugin

Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.

License 许可
Apache-2.0
最近更新
2026-10-08
Stars
★ 0
Forks
0

01 前言 Front Matter

name
nemo-evaluator-plugin
license
Apache-2.0
metadata
{ "owner": "nemo-platform", "maturity": "active" }
description
Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.

02 资源文件 12

03 SKILL.md 正文

Evaluator Plugin

Use this skill for evaluation tasks against a running NeMo Platform server. The plugin-backed CLI interface is nemo evaluator; the legacy generated nemo evaluation API command group is not the target surface for new guidance.

CLI Interface

Prerequisites

  • all commands in this file assume that the shell's working dir is at the root of the Nvidia-NeMo/nemo-platform repo
  • activate the Python virtual environment before invoking the nemo CLI: source .venv/bin/activate

Check plugin status from the CLI:

nemo evaluator info

Metric Types

Explore Available Metrics

To view available metric names, run:

nemo evaluator metric-types

To view a specific metric schema, pass a metric name from the metric_types list above:

nemo evaluator metric-types <metric-name>

Inspect all the registered metric schema contracts:

nemo evaluator evaluate explain

Note: use nemo evaluator evaluate explain as the source of truth for the current plugin input schema. It will return a large json schema response, so strongly prefer nemo evaluator metric-types when you only need metric names and corresponding schemas.

Evaluation Spec

Evaluation spec is a payload that is provided to CLI as an input to execute evaluation.

At a high level, a spec describes:

  • metrics: bundled Evaluator SDK metric configurations
  • dataset: inline rows to evaluate or platform FilesetRef that contains the dataset
  • params: optional Evaluator SDK execution parameters
  • target: optional model or agent target for online evaluation

See the LLM-judge spec example at assets/specs/llm_as_judge.json.

Metric Bundle Payloads

The checked-in spec examples use bundled SDK metrics. The fields under metrics[*].payload are generated by bundle_metric(metric, CloudpickleMetricBundlePackager()).

To see the pattern for configuring a pre-defined SDK metric, for example ExactMatchMetric, and converting it into bundled metric JSON, inspect build_metric_bundle_example() in generate_example_specs.py and run:

uv run --frozen python skills/nemo-evaluator-plugin/scripts/generate_example_specs.py

Run Evaluations

Run Using File Spec Reference

When using the nemo evaluator evaluate run command, results are saved into local temporary directories and the link is printed to stdout. Prefer the --spec-file named argument over inline shell JSON because metric bundles include serialized payloads. Examples of various specs are provided in the assets/specs directory.

Evaluate using exact-match metric

See the spec example at assets/specs/exact_match_metric.json.

nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json

Evaluate using a benchmark metric set

nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_benchmark.json

Evaluate using LLM-Judge metric

Uses an LLM to score responses. See the spec example at assets/specs/llm_as_judge.json.

nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/llm_as_judge.json

Run Evaluation As A Durable Job

Use the nemo evaluator evaluate submit command to create a durable evaluation job. The response of this command returns a job handler object instead of the evaluation result.

nemo evaluator evaluate submit \
  --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json

The submit response includes the generated job's name field, for example nemo-evaluator-zlhn1ecd. Wait for the job to complete, then list and download the job results.

nemo jobs get-status <job-name>
nemo jobs get <job-name>
nemo jobs results list <job-name>
nemo jobs results download aggregate-scores --job <job-name> --output-file aggregate-scores.json
nemo jobs results download row-scores --job <job-name> --output-file row-scores.jsonl

Python SDK Interface

Evaluator Python SDK client is exposed as evaluator variable on NeMoPlatform instance:

from nemo_platform import NeMoPlatform

platform_client = NeMoPlatform(base_url="http://localhost:8080")
status = platform_client.evaluator.plugin_status()

See examples of using the plugin SDK interface in plugin_sdk_examples.py.

Security

Make sure not to print any secrets to stdout since this can be collected as logs

Additional Resources

For LLM-judge setup notes, see LLM Judge Notes.

For evaluator API key auth, see Evaluator API Auth.

For local and cluster troubleshooting, see Evaluation Troubleshooting.

04 同仓库更多 Skills

Solizardking
Solizardking/train2earn

testing

For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.

★ 0
Solizardking
Solizardking/train2earn

qedgen

Formally verify programs by writing Lean 4 proofs. Trigger this skill whenever the user wants to formally verify code, generate Lean 4 proofs, prove properties about algorithms or smart contracts, verify invariants, convert program logic into formal specifications, or anything involving Lean 4 and formal verification. Also trigger when the user mentions "qedgen", "lean proof", "formal proof", "verify my code", "prove correctness", "formal verification", or wants mathematical guarantees about their implementation.

★ 0
Solizardking
Solizardking/train2earn

solana-dev

Use when user asks to "build a Solana dapp", "write an Anchor program", "create a token", "debug Solana errors", "set up wallet connection", "test my Solana program", "deploy to devnet", or "explain Solana concepts" (rent, accounts, PDAs, CPIs, etc.). Also use for quick on-chain lookups via public RPC + curl — "what's the balance of <wallet>", "look up transaction <sig>", "token balance for <account>", "check this address on mainnet/devnet". End-to-end Solana development playbook covering wallet connection, Anchor/Pinocchio programs, Codama client generation, LiteSVM/Mollusk/Surfpool testing, security checklists, and JSON-RPC curl lookups against public clusters. Integrates with the Solana MCP server for live documentation search. Prefers framework-kit (@solana/client + @solana/react-hooks) for UI, wallet-standard-first connection (incl. ConnectorKit), @solana/kit for client/RPC code, and @solana/web3-compat for legacy boundaries.

★ 0
Solizardking
Solizardking/train2earn

solana-common-errors

Diagnose and fix common Solana development errors — GLIBC mismatches, Anchor version conflicts, cargo build-sbf / platform-tools failures, LiteSVM issues, edition2024 crate pins, RPC airdrops, and Anchor 0.29→0.32 migrations. Use when builds fail, Anchor CLI won't install, or CI breaks on Debian/Ubuntu.

★ 0
Solizardking
Solizardking/train2earn

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

★ 0
Solizardking
Solizardking/train2earn

phantom-wallet-mcp

Execute wallet operations through the Phantom MCP server. Use when the user wants to interact with their Phantom wallet directly — get addresses, transfer SOL or SPL tokens, buy/swap tokens, sign transactions, and sign messages across Solana, Ethereum, Bitcoin, and Sui. Requires the @phantom/mcp-server to be configured as an MCP server.

★ 0