skill-tester
Test and benchmark Codex skills empirically using evaluation-driven development. Use when validating a skill's effectiveness, running evals, comparing skill vs. baseline performance, running benchmarks with timing/token metrics, or iterating on skill improvements based on empirical data. Supports both a fast pass/fail check and a full baseline-comparison benchmark with timing/token metrics.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- skill-tester
- description
- Test and benchmark Codex skills empirically using evaluation-driven development. Use when validating a skill's effectiveness, running evals, comparing skill vs. baseline performance, running benchmarks with timing/token metrics, or iterating on skill improvements based on empirical data. Supports both a fast pass/fail check and a full baseline-comparison benchmark with timing/token metrics.
- allowed-tools
- Read Write Edit Glob Grep Agent Bash(python:*) Skill
02 资源文件 9
- 资产文件assets/benchmark-example.json1.4 KB
- 资产文件assets/benchmark-schema.json1.6 KB
- 资产文件assets/eval-metadata-example.json977 B
- 资产文件assets/evals-example.json1.1 KB
- 资产文件assets/grading-baseline-example.json1.0 KB
- 资产文件assets/grading-with-skill-example.json1.2 KB
- 参考文档references/eval-schema.md9.3 KB
- 参考文档references/workflow.md14.0 KB
- 脚本scripts/aggregate_benchmark.py9.7 KB
03 SKILL.md 正文
Skill Tester
Purpose: Empirically validate Codex skills through evaluation-driven testing. Proves skills actually help Codex (with data) rather than guessing.
Mindset
Skills must be measured, not assumed. This pipeline provides systematic evidence: Does the skill improve Codex's performance? By how much? What should we improve next?
When to Use
- Validating a newly created skill against baseline Codex performance without it
- Benchmarking skill refinements across multiple iterations
- Proving skill effectiveness with timing, token usage, and pass rates side-by-side
- Making data-driven decisions on what to improve next in a skill
When NOT to Use
- Creating skills — use
skill-developmentinstead. - Reviewing skill quality — use
skill-reviewerfor a single-pass structural quality check without benchmarking. - Validating plugin structure — use the
plugin-validatoragent for manifest and directory checks. - Testing during initial skill authoring (Phase 3 of skill creation, before the skill is finalized) — use
skill-development's own Phase 3 workflow; switch to this skill once the skill exists and you need a dedicated with/baseline benchmark or a multi-iteration comparison. - Automated structural fix-review loops until skill-reviewer passes — use
skill-improver-loopinstead; this skill iterates on empirical eval/benchmark data, not skill-reviewer's structural findings.
Core Principles
Empirical Evidence — Intuition ≠ proof. Collect actual performance data: pass rates, tokens, timing.
Parallel Testing — For each eval, test WITH skill AND baseline SIMULTANEOUSLY (2 agents per eval, in parallel). Eliminates confounding variables from different test runs.
Workspace Isolation — Each iteration lives in workspace/iteration-N/. Keeps history clean. Easy to compare iteration 1 → 2 → 3 performance.
Schema Consistency — All JSON outputs follow strict schemas (evals.json, grading.json, benchmark.json, timing.json). Enables reliable aggregation and comparison.
Quick Start: The 7-Phase Pipeline
Phase 1: Setup → Identify skill + confirm what it does
Phase 2: Create Evals → Interview user → write test cases + assertions
Phase 3: Run Tests → Launch 2 agents per eval (with_skill + baseline) in parallel
Phase 4: Grade Results → Evaluate outputs against assertions
Phase 5: Aggregate → Run aggregation script to compute benchmark.json
Phase 6: Review Summary → Show comparison table + improvement suggestions
Phase 7: Iterate → Update skill + run next iteration (or stop if satisfied)
Artifact Location (Standardized)
All evaluation artifacts live in a centralized ./evals/ directory at project root.
./evals/ ← Project root, NOT inside skill directory
├── <skill-name-1>/
│ ├── evals.json
│ └── workspace/
│ ├── iteration-1/
│ │ ├── eval-1/, eval-2/, eval-3/ ← Per-eval directories
│ │ └── benchmark.json ← Aggregated results
│ └── iteration-2/
│ └── ...
└── <skill-name-2>/
└── ...
See references/workflow.md — "Phase 1: Setup" for workspace location rationale.
PHASE 1: Setup
Goal: Identify the skill being tested. Confirm what it does. Prepare workspace.
Step 1.1: Ask User - Which Skill?
Use AskUserQuestion to ask which skill to test:
question: "Which skill do you want to test?"
header: "Skill Selection"
options: [
{label: "skill-development", description: "Testing skill-development from skills/ directory"},
{label: "skill-refiner-interactive", description: "Testing skill-refiner-interactive from skills/ directory"},
{label: "Other skill", description: "Testing a skill not listed above"}
]
⏸️ Collect user input. Wait for response before proceeding.
Step 1.2: Confirm Skill Path & Purpose
Read the skill's SKILL.md to understand:
- What does this skill do?
- What are its core use cases?
- What problem does it solve?
Show user a summary:
Skill: <name>
Location: <path>
Purpose: <one-line summary from description>
Step 1.2b: Choose Workflow Mode
Question: Which testing mode do you want?
Use AskUserQuestion with 2 options:
questions: [
{
question: "Which testing mode would you like?",
header: "Workflow Mode",
options: [
{
label: "Quick Workflow",
description: "Fast validation: only test WITH skill (no baseline), no timing metrics. Just pass/fail on assertions. Perfect for quick checks."
},
{
label: "Full Pipeline",
description: "Complete analysis: test WITH skill + baseline parallel, measure tokens/timing, aggregate & compare. For comprehensive benchmarking."
}
],
multiSelect: false
}
]
⏸️ Collect user input.
If Quick Workflow selected:
- Skip to Phase 2 with note: "Quick mode: only with_skill, no baseline comparison"
- Jump to simplified Phase 3 (only 1 agent per eval)
- Skip Phase 5 (aggregation) and Phase 6 comparison tables
- Show simple pass/fail results
If Full Pipeline selected:
- Continue with standard 7-phase pipeline (proceed to Step 1.3)
Step 1.3: Create Workspace
Create directory structure in project root (standardized):
./evals/<skill-name>/
./evals/<skill-name>/workspace/iteration-1/
Log workspace path. All subsequent phases write to this directory (NOT inside the skill's directory).
PHASE 2: Create Evals
Goal: Build evaluation cases that measure skill effectiveness.
Step 2.1: Interview User for Test Scenarios
Ask 3 questions plus an optional 4th (use AskUserQuestion with free-form "Other" option):
Question 1: "What are 2-3 core scenarios this skill should handle?"
Example: "skill-development should handle: creating a new skill from scratch,
converting a slash command, improving an existing skill"
Question 2: "For each scenario, what makes a GOOD response?"
Example: "Good responses: skill has clear name, description with trigger phrases,
correct SKILL.md structure, efficient references/"
Question 3: "What should FAIL (baseline without skill)?"
Example: "Baseline will miss best practices, create vague descriptions,
skip necessary structure"
Question 4 (optional): "Should any of these scenarios also be tested under pressure —
e.g. time constraints, sunk cost, exhaustion, authority — to verify the skill holds up
when an agent is incentivized to skip it?"
Example: "Yes — the validation scenario should be re-run with a 'you have 5 minutes' framing"
⏸️ Collect responses. Store in memory.
If Question 4 is answered yes, this pipeline's quantitative pass rates don't cover that axis — run skill-development's Phase 3.5 compliance testing (references/compliance-testing.md) before or alongside this pipeline; see "Integration with skill-development, skill-refiner-interactive" in references/workflow.md.
Step 2.1b: Cross-Check Against the Target's Own Testing & Validation Section
Before writing evals.json, Read the target skill's own SKILL.md for a "Testing & Validation" section. If one exists and lists numbered scenarios, check whether the eval scenarios collected in Step 2.1 collectively exercise each item — or explicitly note the gap for any that aren't covered. A skill's own Testing & Validation checklist is a direct, authoritative claim about what "tested" should mean for it; writing an eval set that doesn't cross-check against it risks shipping eval coverage narrower than the skill's own documented claims, which reads as tested when it isn't.
Step 2.2: Generate evals.json
Write ./evals/<skill-name>/evals.json with fields: skill_name, skill_path, description, and evals array (each entry: id, name, prompt, expected_output, files).
See references/eval-schema.md for full schema and annotated examples.
Step 2.3: Generate eval_metadata.json Files
Create workspace/iteration-1/eval-N/eval_metadata.json for each eval. Required fields: eval_id, skill_path, assertions array (each entry: text, type, target). Assertion types: presence, quality, structure, functionality.
Create directories: workspace/iteration-1/eval-1/, eval-2/, etc. (with_skill and baseline subdirs created in Phase 3).
See references/eval-schema.md for full schema and annotated examples.
Step 2.4: Plugin-Rule Compliance Assertions (Optional)
When evaluating a skill for plugin-rule compliance (naming, tool-scoping, language, formatting), invoke plugin-rulebook before writing eval_metadata.json to obtain the active rule set. Use each enabled rule as a separate assertion:
Skill: plugin-rulebook
→ Returns active rules with descriptions, examples, and enforcement levels
Add rule-compliance assertions to eval_metadata.json alongside functional assertions:
{"text": "allowed-tools uses space-separated format (R6)", "type": "structure", "target": "SKILL.md frontmatter"}
QUICK WORKFLOW (Alternative Path)
If user chose "Quick Workflow" in Step 1.2b, follow this streamlined path instead of Phases 3-7.
Quick Phase 1: Run WITH_SKILL Only (No Baseline)
Goal: Test the skill itself. No baseline comparison, no timing metrics. Just: Does it work?
For EACH eval in evals.json, launch ONE agent (no parallel baseline):
See references/eval-schema.md — "Agent Prompt Templates" section for the WITH_SKILL_ONLY prompt template (Quick Workflow variant, no timing metrics).
⏸️ Wait for agent completion before proceeding to next eval.
Quick Phase 2: Grade Results (Assertions Only)
For each eval, grade against assertions in eval_metadata.json. Record pass/fail + evidence for each assertion.
Save to: ./evals/<skill-name>/workspace/iteration-1/eval-N/with_skill/grading.json
See references/eval-schema.md for grading.json schema.
Quick Phase 3: Show Results
Display simple pass/fail summary (no benchmark.json):
QUICK VALIDATION RESULTS: <skill-name>
====================================
Eval 1: <Scenario> ✓ PASS (5/5 assertions)
Eval 2: <Scenario> ✓ PASS (4/5 assertions)
Eval 3: <Scenario> ✗ FAIL (2/5 assertions)
Summary: 11/15 assertions passed (73%)
Status: Ready to refine or deploy
Quick Phase 4: Next Steps
Ask user:
question: "What would you like to do?"
header: "Next Steps"
options: [
{label: "Run full pipeline", description: "Move to complete benchmarking with baseline comparison"},
{label: "Refine skill", description: "Update skill based on failed assertions"},
{label: "Done", description: "Quick validation complete"}
]
PHASE 3: Run Tests
Goal: Execute 2 agents per eval (WITH skill + BASELINE) in parallel. Capture outputs.
Step 3.1: Spawn Agents (Parallel Execution)
For EACH eval, launch 2 agents SIMULTANEOUSLY in one Agent tool call:
See references/eval-schema.md — "Agent Prompt Templates" section for the WITH_SKILL and BASELINE prompt templates.
⏸️ Wait for both agents to complete before proceeding to Phase 4.
PHASE 4: Grade Results
Goal: Evaluate each agent's outputs against assertions in eval_metadata.json.
Step 4.1: Review Outputs & Grade
For EACH eval:
- Read
with_skill/outputs/files - Read
baseline/outputs/files - For EACH assertion in
eval-N/eval_metadata.json:- Does with_skill output satisfy the assertion? (true/false)
- Does baseline output satisfy the assertion? (true/false)
- Collect brief evidence quotes from outputs
Step 4.2: Write grading.json Files
Create workspace/iteration-1/eval-N/with_skill/grading.json and baseline/grading.json.
Required fields: eval_id, configuration ("with_skill" or "baseline"), assertions_evaluated array (each: text, passed, evidence), summary (assertions_total, assertions_passed, pass_rate).
See references/eval-schema.md for full schema and pass/fail examples.
PHASE 5: Aggregate
Goal: Compute benchmark.json with summary stats (pass rates, tokens, timing).
Step 5.1: Run Aggregation Script
Execute the aggregation script (see scripts/aggregate_benchmark.py for the full implementation; references/eval-schema.md for the invocation command and output schema):
python ${CLAUDE_SKILL_DIR}/scripts/aggregate_benchmark.py \
./evals/<skill-name>/workspace/iteration-1
Example:
python ${CLAUDE_SKILL_DIR}/scripts/aggregate_benchmark.py \
./evals/skill-development/workspace/iteration-1
This reads all grading.json and timing.json files from the standardized location, outputs benchmark.json with per-eval stats (pass rates, tokens, duration, delta) and summary statistics.
See references/eval-schema.md for benchmark.json schema and full example.
PHASE 6: Review Summary
Goal: Show user a clear comparison table + next steps.
Step 6.1: Render Comparison Table
Display results in human-readable format:
EVALUATION RESULTS: skill-tester iteration-1
==============================================
Eval 1: [Scenario Name]
WITH SKILL: 3/3 (100%) | 2500 tokens | 8s
BASELINE: 2/3 (67%) | 1800 tokens | 5s
DELTA: +33% better | +700 tokens | +3s
Eval 2: [Scenario Name]
WITH SKILL: 2/3 (67%) | 2100 tokens | 7s
BASELINE: 2/3 (67%) | 1900 tokens | 4s
DELTA: Even | +200 tokens | +3s
SUMMARY
-------
With Skill Avg: 96.5% pass rate | 2300 avg tokens
Baseline Avg: 67.0% pass rate | 1850 avg tokens
IMPROVEMENT: +29.5 percentage points
Step 6.2: Offer Next Steps
Ask user (AskUserQuestion):
question: "What would you like to do?"
header: "Next Steps"
options: [
{
label: "Iterate (update skill, run next iteration)",
description: "Based on results, modify the skill and run workspace/iteration-2"
},
{
label: "Stop (satisfied with results)",
description: "Evaluation complete. Save benchmark data for reference"
},
{
label: "Refine evals (change test cases, rerun)",
description: "Update evals.json and run iteration again with same eval set"
}
]
⏸️ Collect user input.
PHASE 7: Iterate
Goal: If user chose "Iterate", update the skill and run next iteration.
Step 7.1: Get Improvement Feedback
Ask user (AskUserQuestion, free-form):
question: "What would you like to improve about the skill?"
header: "Improvement"
⏸️ Collect feedback.
Step 7.2: Update Skill
Use skill-refiner-interactive or direct edits to improve the skill based on feedback:
- Reword descriptions for clarity
- Add missing guidance to SKILL.md body
- Reorganize references
- Trim token usage
Step 7.3: Run Next Iteration
Create workspace/iteration-2/ directory. Repeat Phases 3–6 with updated skill.
Step 7.4: Compare Iterations
Show delta between iteration-1 benchmark and iteration-2 benchmark:
ITERATION COMPARISON
====================
Iteration 1 pass rate: 67% → Iteration 2: 95% (+28 points)
Iteration 1 tokens: 1900 → Iteration 2: 2100 (+200, acceptable)
Loop back to Phase 6 (Step 6.2) to ask: iterate again or stop?
Suggested next step: if a benchmark run shows a regression, a failed assertion, or a Quick Workflow FAIL, ask with AskUserQuestion: "Run enhancement-suggestor against these results for a classified (complexity/risk/benefit) WHAT/WHY/HOW action plan?" — options "Yes" / "No". If yes, invoke the enhancement-suggestor agent (via Agent) against the results. Never invoke it without asking first.
Testing & Validation
After a test run, verify:
- Trigger phrases — confirm skill activates on: "run evals on skill-X", "benchmark this skill", "validate skill performance", "compare skill vs. baseline", "test skill effectiveness"
- Non-triggers — confirm skill does NOT activate on: "review my skill structure", "create a new skill", "fix this PR"
- Mode selection — Quick Workflow and Full Pipeline branches both produce correctly structured output directories
- Schema integrity — all JSON files (
evals.json,grading.json,benchmark.json,timing.json,eval_metadata.json) validate againstreferences/eval-schema.mdschemas - Baseline parity — baseline agent receives no SKILL.md content; with_skill agent receives full SKILL.md content
Verified 2026-07-11: items 1 and 2 checked directly against the current frontmatter description — each of the 5 trigger phrases maps to specific description language ("running evals", "validating a skill's effectiveness", "comparing skill vs. baseline performance", "running benchmarks"), and none of the 3 non-trigger phrases share that vocabulary (they map to skill-reviewer's and skill-development's domains instead). aggregate_benchmark.py's new guard/sort behavior (items covering script robustness) was verified with synthetic fixtures — see the script's own commit history. A live end-to-end pipeline run (spawning real with_skill/baseline agents) has not been performed — that remains the one unverified item.
Quality gates:
- [ ] WITH_SKILL and BASELINE agents launched simultaneously (not sequentially) in Full Pipeline
- [ ]
benchmark.jsonexists inworkspace/iteration-N/before presenting Phase 6 summary - [ ] All destructive operations (deleting eval dirs) confirmed with user before execution
Reference Guide
| Resource | Purpose |
|---|---|
references/eval-schema.md |
JSON schemas, agent prompt templates, and workspace structure |
references/workflow.md |
Decision points and detailed workflow guidance |
scripts/aggregate_benchmark.py |
Python script that aggregates grading/timing data into benchmark.json |
plugin-rulebook skill |
Active rule configuration for compliance-testing assertions |
enhancement-suggestor agent |
Turns a regression/fail/delta result into a classified, prioritized action plan |
04 同仓库更多 Skills
demo
Fixture skill used by scripts.marketplace_ci sync tests.
export-demo
Fixture skill used by scripts.marketplace_ci export tests.
work-to-development
Validate an accepted Linear Issue's readiness for implementation (criteria, dependencies, duplicate/existing artifacts) and request governed branch/worktree creation through git-kit:starting-work — recording work-started Git/GitHub evidence and a deliberate Linear Started transition only after Git identity is read back and verified. Use when asked to start work on a Linear issue, begin development for an accepted issue, or prepare an accepted issue for implementation. Never creates a branch directly — always delegates to git-kit.
work-linking
Create and maintain stable cross-system links and authority labels between Notion and Linear records, and detect/repair drift between them. Use when asked to link a Notion record to a Linear record, check for drift between Notion and Linear, or repair a broken/stale link. Repairs only the bounded non-authoritative copy or link — never chooses by newest timestamp, and never creates a reverse-write loop against either system's own authority.
status-and-learning
Produce a dated, explicitly non-live Linear-to-Notion progress summary, or capture an outcome/learning after work completes. Use when asked to summarize progress to Notion, report portfolio status, or capture what was learned from a completed piece of work — not to disposition open follow-up items (see open-item-management for that). Never runs automatically or continuously — every summary is a deliberate, approved snapshot, not a live sync.
repository-gates
Discover, delegate, record, and invalidate this repository's actual Git/GitHub lifecycle gates (pre-commit, pre-push, PR required checks, Codex delta review, review requirements, merge rights) and the repository-policy provider profile that maps governed operations to git-kit. Use when asked what gates apply to this repository, to discover repository policy before starting Wave 2 work, or to check whether prior gate evidence is still valid after new commits. Discovery and delegation only — never executes a gate's own internal logic, and never invents a gate the repository doesn't actually define (in particular, never assumes a "Review Changes" gate exists).