AGENT SKILL

plugin-evals

Build and run an eval suite for a plugin — realistic prompts, graders, repeated runs, and a no-plugin baseline — and gate changes on the scores. Use when the user asks to "test my plugin", "write evals", "benchmark a plugin", "measure whether a skill triggers", or before shipping plugin changes.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 21
Forks
2

01 前言 Front Matter

name
plugin-evals
description
Build and run an eval suite for a plugin — realistic prompts, graders, repeated runs, and a no-plugin baseline — and gate changes on the scores. Use when the user asks to "test my plugin", "write evals", "benchmark a plugin", "measure whether a skill triggers", or before shipping plugin changes.

03 SKILL.md 正文

Plugin Evals

Measure what a plugin reliably does. A case is a realistic prompt plus one or more graders — pass/fail checks on the final reply, the transcript, tool calls, or created files. Runs are repeated and averaged, and every case also runs without the plugin so the scores show the plugin's contribution, not the model's baseline ability.

Suite layout

The suite lives under evals/ in the plugin root. One directory per case; nest cases under a plain directory to group them.

<plugin>/
├── plugin.json
├── skills/...
└── evals/
    ├── <case>/
    │   ├── prompt.md          # the prompt sent to the agent, plus run limits
    │   ├── case.yaml          # optional: workspace fixtures, history, extra dirs
    │   ├── graders/
    │   │   ├── criteria.md    # one grader per file
    │   │   └── skill-fired.md
    │   └── <fixtures, scripts referenced by case.yaml>
    ├── mocks/                 # optional: scripted MCP tool answers
    └── results/               # written by each run; add to .gitignore

Only prompt.md or case.yaml is required for a case to exist. Cases start from an empty working directory, so put everything the task needs in the prompt, in case.yaml fixtures, or in a fixture directory added via context.add_dirs.

Step 1 — Design the cases

Interview the user with ask_user_question:

  1. Which skill, command, agent or MCP server the suite tests.
  2. What a good result looks like, concretely — the artifact, its shape, the tool that must have run.
  3. What should NOT trigger the plugin (negative cases: a near-miss request the skill should leave alone).
  4. Whether the task needs workspace fixtures (a git repo, sample files) or an early conversation to continue.

Write the prompt the way a user would type it, phrased naturally — not naming the skill. A prompt that names the skill measures obedience, not triggering.

prompt.md frontmatter fields:

Field Default Purpose
name directory name Case name; report keys and --case filters use it
description — For humans; unused at run time
tags [] Filter with --tag
runs 3 Runs per arm, 1–50
model default The agent under test's model
max_turns 10 Turn cap; hitting it is recorded as a run error
timeout_seconds 300 Wall-clock cap per run
allowed_tools [] Tools the case may use, e.g. [Read, Glob, Grep, Skill]
env {} Extra environment variables, keys must match EVAL_[A-Z0-9_]*

case.yaml (needs schema_version: "1.1" and name) adds the fields only it carries: context.scaffold_script (a bash script in the case directory that seeds the workspace before the agent starts), context.history_file (a .jsonl transcript to continue; the prompt becomes the next user turn), context.add_dirs (read-only fixture directories), and execution.prompt when the whole case lives in YAML.

Step 2 — Write the graders

One grader per file under graders/. Frontmatter sets type plus options; optional weight (relative weight in the run score) and arm (with-only excludes it from scoring in the baseline arm).

Type Options Passes when
regex pattern, flags, match, target JavaScript regex found in the target; match: not_contains requires absence, match: "count:N" requires exactly N
tool_used tool, input_match, min, max Calls to tool whose JSON input matches input_match land between min (1) and max (∞); min: 0, max: 0 asserts the tool was never called
tool_order before, after Both tools ran and the first before call precedes the first after call
file_exists path, exists A file created during the run matches the glob (or none does with exists: false)
llm criteria, focus A judge model votes PASS on the rubric at least 2 of 3 votes
baseline baseline_file, criteria The judge finds the run at least as good as a reference .jsonl transcript

regex and llm accept target/focus: last_message (default), trace (whole session as JSON lines), files (created paths), { source: file, path: <path> } (a file's content — grade generated artifacts with this, never with files), or mock_calls.

The skill-triggered check, for a skill named your-skill:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill"'
---

Graders that give a stable signal

  • Grade long outputs (generated files) with regex over { source: file, path }; keep llm graders for short outputs with concrete PASS and FAIL conditions.
  • Give every case one grader about the outcome and one about how the result was produced (tool_used or tool_order). Together they say whether the answer is right and whether the plugin produced it.
  • If the tool_used: Skill grader passes but Δ is negative, suspect the judge: rerun with a stronger judge model and tighten the rubric so formatting does not decide the verdict.

Step 3 — Baseline and arms

Every case runs in two arms: with the plugin and without any plugin (same run counts). The report shows WITH, W/OUT and their difference Δ — the plugin's contribution. A case scoring 1.0 in both arms proves nothing about the plugin.

Graders that cannot pass without the plugin (every tool_used: Skill grader, and any grader marked arm: with-only) are excluded from scoring in the without-arm so the arms stay comparable. Mark a grader arm: both to force scoring in both arms — that is what a "must NOT call the skill" check wants, with min: 0 and max: 0.

Step 4 — Mock MCP servers

Skills that call MCP tools run without the real services. Place evals/mocks/<server>/<tool>.md suite-wide, or mocks/ inside a case. The file body is the tool's answer; {{input.<field>}} inserts call input, {{file:fixtures/<name>}} inserts a fixture file, and an expect: block of dot-path → type guards the input (a violating call aborts the run with score 0). error: true returns the body as a tool error. A server with no mock files never starts.

Grade the calls themselves with a grader whose target is mock_calls.

Step 5 — Run on DeepSeek Harness

There is no bundled eval command here; the same suite format runs through the agent, with the host doing what such a CLI would:

  1. Isolation. One fresh session per run, loading only the plugin under test — no personal or project-level skills, hooks, MCP servers, or memory. Point the run at a clean DSH_HOME and a scratch working directory.
  2. Spawn. For each case × run × arm, start the run (subagent_run) with the prompt body from prompt.md, the case's allowed tools, and the turn/timeout caps from the frontmatter. Launch all runs for a turn in one message so they run concurrently.
  3. Observe triggering from the session log. A skill load is an ordinary tool call — {"type":"tool/call","data":{"name":"skill","arguments":"{\"name\":\"<skill>\"}"}} — recorded in the subagent's persisted session log at $DSH_HOME/sessions/.... The skill-creator skill's scripts/run_eval.py reads these logs, detects the call, and builds the confusion matrix; reuse it instead of re-parsing logs by hand.
  4. Grade. Apply the deterministic graders to the captured reply, transcript and files. For llm and baseline graders, spawn judge subagents with the rubric and take the 2-of-3 majority. Record per-run per-grader verdicts plus cost and wall time.
  5. Aggregate. Run score = weighted share of passing graders; case score = mean over runs; a case passes at --threshold (default 1.0). Render the summary table:
CASE        WITH  W/OUT Δ      RUNS COST    NOTES
first-case  1.00  0.33  +0.67  6    ...

NOTES shows the highest-weight failing grader's explanation, or the with-arm run error.

  1. Iterate. The most common first finding is Δ near zero with a failed tool_used: Skill grader: the description does not trigger on the prompt's wording. Adjust the description, rerun, compare. To iterate one case cheaply, run one arm once (--runs 1, no baseline) — noisy, so confirm at three runs before trusting any change.

Step 6 — Gate in CI

Fail the check when any case score falls below the threshold the team sets. Add evals/results/ to .gitignore; commit the cases and mocks. Rerun the suite when the plugin changes or a new model ships — both are regression sources. If a run dies on a rate or usage limit mid-suite, treat its scores as suspect and rerun after the limit resets.

Step 7 — Report

Report the summary table, the report directory, the per-case verdicts against the threshold, and the descriptions or skills whose triggering needs another iteration.

04 同仓库更多 Skills

Sivan757
Sivan757/dsh-agent-plugins-market

verify

Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.

★ 21
Sivan757
Sivan757/dsh-agent-plugins-market

simplify

Review changed code for reuse, quality, and efficiency, then fix any issues found. Use when the user asks to "simplify the code", "clean up the diff", "reduce duplication in my changes", or after finishing a feature before committing.

★ 21
Sivan757
Sivan757/dsh-agent-plugins-market

insight

Analyze this machine's DeepSeek Harness session history and report how sessions actually go — operating style, what works, friction points, and what to try next. Use when the user asks "how have I been using this", "analyze my sessions", "make an insight report", or wants a retrospective.

★ 21
Sivan757
Sivan757/dsh-agent-plugins-market

init

Set up a minimal AGENTS.md (and optionally skills and hooks) for this repo. Use when the user asks to "initialize the project", "generate AGENTS.md", "set up project instructions", or runs /init.

★ 21
Sivan757
Sivan757/dsh-agent-plugins-market

explore

Fast read-only codebase exploration that returns a report instead of filling the main context. Use when the user asks to "explore the codebase", "find where X is handled", "how does Y work", or when a search would take many reads to answer.

★ 21
Sivan757
Sivan757/dsh-agent-plugins-market

automation-recommender

Analyze a codebase and recommend DeepSeek Harness automations (hooks, subagents, skills, plugins, MCP servers). Use when user asks for automation recommendations, wants to optimize their DeepSeek Harness setup, mentions improving DeepSeek Harness workflows, asks how to first set up DeepSeek Harness for a project, or wants to know what DeepSeek Harness features they should use.

★ 21