plugin-evals
Build and run an eval suite for a plugin — realistic prompts, graders, repeated runs, and a no-plugin baseline — and gate changes on the scores. Use when the user asks to "test my plugin", "write evals", "benchmark a plugin", "measure whether a skill triggers", or before shipping plugin changes.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 21
- Forks
- 2
01 前言 Front Matter
- name
- plugin-evals
- description
- Build and run an eval suite for a plugin — realistic prompts, graders, repeated runs, and a no-plugin baseline — and gate changes on the scores. Use when the user asks to "test my plugin", "write evals", "benchmark a plugin", "measure whether a skill triggers", or before shipping plugin changes.
03 SKILL.md 正文
Plugin Evals
Measure what a plugin reliably does. A case is a realistic prompt plus one or more graders — pass/fail checks on the final reply, the transcript, tool calls, or created files. Runs are repeated and averaged, and every case also runs without the plugin so the scores show the plugin's contribution, not the model's baseline ability.
Suite layout
The suite lives under evals/ in the plugin root. One directory per case; nest cases under a plain directory to group them.
<plugin>/
├── plugin.json
├── skills/...
└── evals/
├── <case>/
│ ├── prompt.md # the prompt sent to the agent, plus run limits
│ ├── case.yaml # optional: workspace fixtures, history, extra dirs
│ ├── graders/
│ │ ├── criteria.md # one grader per file
│ │ └── skill-fired.md
│ └── <fixtures, scripts referenced by case.yaml>
├── mocks/ # optional: scripted MCP tool answers
└── results/ # written by each run; add to .gitignore
Only prompt.md or case.yaml is required for a case to exist. Cases start from an empty working directory, so put everything the task needs in the prompt, in case.yaml fixtures, or in a fixture directory added via context.add_dirs.
Step 1 — Design the cases
Interview the user with ask_user_question:
- Which skill, command, agent or MCP server the suite tests.
- What a good result looks like, concretely — the artifact, its shape, the tool that must have run.
- What should NOT trigger the plugin (negative cases: a near-miss request the skill should leave alone).
- Whether the task needs workspace fixtures (a git repo, sample files) or an early conversation to continue.
Write the prompt the way a user would type it, phrased naturally — not naming the skill. A prompt that names the skill measures obedience, not triggering.
prompt.md frontmatter fields:
| Field | Default | Purpose |
|---|---|---|
name |
directory name | Case name; report keys and --case filters use it |
description |
— | For humans; unused at run time |
tags |
[] |
Filter with --tag |
runs |
3 |
Runs per arm, 1–50 |
model |
default | The agent under test's model |
max_turns |
10 |
Turn cap; hitting it is recorded as a run error |
timeout_seconds |
300 |
Wall-clock cap per run |
allowed_tools |
[] |
Tools the case may use, e.g. [Read, Glob, Grep, Skill] |
env |
{} |
Extra environment variables, keys must match EVAL_[A-Z0-9_]* |
case.yaml (needs schema_version: "1.1" and name) adds the fields only it carries: context.scaffold_script (a bash script in the case directory that seeds the workspace before the agent starts), context.history_file (a .jsonl transcript to continue; the prompt becomes the next user turn), context.add_dirs (read-only fixture directories), and execution.prompt when the whole case lives in YAML.
Step 2 — Write the graders
One grader per file under graders/. Frontmatter sets type plus options; optional weight (relative weight in the run score) and arm (with-only excludes it from scoring in the baseline arm).
| Type | Options | Passes when |
|---|---|---|
regex |
pattern, flags, match, target |
JavaScript regex found in the target; match: not_contains requires absence, match: "count:N" requires exactly N |
tool_used |
tool, input_match, min, max |
Calls to tool whose JSON input matches input_match land between min (1) and max (∞); min: 0, max: 0 asserts the tool was never called |
tool_order |
before, after |
Both tools ran and the first before call precedes the first after call |
file_exists |
path, exists |
A file created during the run matches the glob (or none does with exists: false) |
llm |
criteria, focus |
A judge model votes PASS on the rubric at least 2 of 3 votes |
baseline |
baseline_file, criteria |
The judge finds the run at least as good as a reference .jsonl transcript |
regex and llm accept target/focus: last_message (default), trace (whole session as JSON lines), files (created paths), { source: file, path: <path> } (a file's content — grade generated artifacts with this, never with files), or mock_calls.
The skill-triggered check, for a skill named your-skill:
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill"'
---
Graders that give a stable signal
- Grade long outputs (generated files) with
regexover{ source: file, path }; keepllmgraders for short outputs with concrete PASS and FAIL conditions. - Give every case one grader about the outcome and one about how the result was produced (
tool_usedortool_order). Together they say whether the answer is right and whether the plugin produced it. - If the
tool_used: Skillgrader passes but Δ is negative, suspect the judge: rerun with a stronger judge model and tighten the rubric so formatting does not decide the verdict.
Step 3 — Baseline and arms
Every case runs in two arms: with the plugin and without any plugin (same run counts). The report shows WITH, W/OUT and their difference Δ — the plugin's contribution. A case scoring 1.0 in both arms proves nothing about the plugin.
Graders that cannot pass without the plugin (every tool_used: Skill grader, and any grader marked arm: with-only) are excluded from scoring in the without-arm so the arms stay comparable. Mark a grader arm: both to force scoring in both arms — that is what a "must NOT call the skill" check wants, with min: 0 and max: 0.
Step 4 — Mock MCP servers
Skills that call MCP tools run without the real services. Place evals/mocks/<server>/<tool>.md suite-wide, or mocks/ inside a case. The file body is the tool's answer; {{input.<field>}} inserts call input, {{file:fixtures/<name>}} inserts a fixture file, and an expect: block of dot-path → type guards the input (a violating call aborts the run with score 0). error: true returns the body as a tool error. A server with no mock files never starts.
Grade the calls themselves with a grader whose target is mock_calls.
Step 5 — Run on DeepSeek Harness
There is no bundled eval command here; the same suite format runs through the agent, with the host doing what such a CLI would:
- Isolation. One fresh session per run, loading only the plugin under test — no personal or project-level skills, hooks, MCP servers, or memory. Point the run at a clean
DSH_HOMEand a scratch working directory. - Spawn. For each case × run × arm, start the run (
subagent_run) with the prompt body fromprompt.md, the case's allowed tools, and the turn/timeout caps from the frontmatter. Launch all runs for a turn in one message so they run concurrently. - Observe triggering from the session log. A skill load is an ordinary tool call —
{"type":"tool/call","data":{"name":"skill","arguments":"{\"name\":\"<skill>\"}"}}— recorded in the subagent's persisted session log at$DSH_HOME/sessions/.... Theskill-creatorskill'sscripts/run_eval.pyreads these logs, detects the call, and builds the confusion matrix; reuse it instead of re-parsing logs by hand. - Grade. Apply the deterministic graders to the captured reply, transcript and files. For
llmandbaselinegraders, spawn judge subagents with the rubric and take the 2-of-3 majority. Record per-run per-grader verdicts plus cost and wall time. - Aggregate. Run score = weighted share of passing graders; case score = mean over runs; a case passes at
--threshold(default 1.0). Render the summary table:
CASE WITH W/OUT Δ RUNS COST NOTES
first-case 1.00 0.33 +0.67 6 ...
NOTES shows the highest-weight failing grader's explanation, or the with-arm run error.
- Iterate. The most common first finding is Δ near zero with a failed
tool_used: Skillgrader: the description does not trigger on the prompt's wording. Adjust the description, rerun, compare. To iterate one case cheaply, run one arm once (--runs 1, no baseline) — noisy, so confirm at three runs before trusting any change.
Step 6 — Gate in CI
Fail the check when any case score falls below the threshold the team sets. Add evals/results/ to .gitignore; commit the cases and mocks. Rerun the suite when the plugin changes or a new model ships — both are regression sources. If a run dies on a rate or usage limit mid-suite, treat its scores as suspect and rerun after the limit resets.
Step 7 — Report
Report the summary table, the report directory, the per-case verdicts against the threshold, and the descriptions or skills whose triggering needs another iteration.
04 同仓库更多 Skills
verify
Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
simplify
Review changed code for reuse, quality, and efficiency, then fix any issues found. Use when the user asks to "simplify the code", "clean up the diff", "reduce duplication in my changes", or after finishing a feature before committing.
insight
Analyze this machine's DeepSeek Harness session history and report how sessions actually go — operating style, what works, friction points, and what to try next. Use when the user asks "how have I been using this", "analyze my sessions", "make an insight report", or wants a retrospective.
init
Set up a minimal AGENTS.md (and optionally skills and hooks) for this repo. Use when the user asks to "initialize the project", "generate AGENTS.md", "set up project instructions", or runs /init.
explore
Fast read-only codebase exploration that returns a report instead of filling the main context. Use when the user asks to "explore the codebase", "find where X is handled", "how does Y work", or when a search would take many reads to answer.
automation-recommender
Analyze a codebase and recommend DeepSeek Harness automations (hooks, subagents, skills, plugins, MCP servers). Use when user asks for automation recommendations, wants to optimize their DeepSeek Harness setup, mentions improving DeepSeek Harness workflows, asks how to first set up DeepSeek Harness for a project, or wants to know what DeepSeek Harness features they should use.