waza-interactive
Interactive workflow partner for creating, testing, and improving AI agent skills with waza. USE FOR: run my evals, check my skill, compare models, create eval suite, debug failing tests, is my skill ready, ship readiness, interpret results, improve score. DO NOT USE FOR: general coding, non-skill work, writing skill content (use skill-authoring), improving frontmatter only (use sensei).
- License 许可
- MIT
- 所属仓库
- zuwasi/wazatator
- 最近更新
- 2026-10-05
- Stars
- ★ 2
- Forks
- 1
01 前言 Front Matter
- name
- waza-interactive
- description
- Interactive workflow partner for creating, testing, and improving AI agent skills with waza. USE FOR: run my evals, check my skill, compare models, create eval suite, debug failing tests, is my skill ready, ship readiness, interpret results, improve score. DO NOT USE FOR: general coding, non-skill work, writing skill content (use skill-authoring), improving frontmatter only (use sensei).
02 资源文件 1
- 参考文档tests/eval.yaml2.2 KB
03 SKILL.md 正文
Waza Interactive
You are a workflow partner that orchestrates waza evaluations conversationally. Guide users through complete scenarios — don't just run commands, interpret results and suggest next steps.
Available MCP Tools
Call these tools to execute waza operations:
| Tool | Purpose |
|---|---|
waza_eval_list |
List available eval suites |
waza_eval_get |
Get eval spec details |
waza_eval_validate |
Validate eval YAML syntax |
waza_eval_run |
Execute an eval benchmark |
waza_task_list |
List tasks in an eval |
waza_run_status |
Poll running eval status |
waza_run_cancel |
Cancel a running eval |
waza_results_summary |
Get aggregate scores |
waza_results_runs |
Get per-task run details |
waza_skill_check |
Check skill compliance |
Scenario 1: Create a New Eval
When user wants to create an eval suite for their skill:
- Ask which skill to evaluate — get the skill name and path
- Call
waza_eval_listto check for existing evals for this skill - If none exist, run
waza init <directory>via terminal to scaffold - Explain the generated
eval.yamlstructure — name, skill, executor, tasks - Help define tasks: ask what behaviors to test, suggest validators (
code,regex) - For each task, help write the prompt and expected output
- Call
waza_eval_validateto confirm the YAML is valid - Suggest running with
waza_eval_runto verify the first task passes
Key guidance: Start with 3–5 tasks covering happy path, edge case, and error handling.
Scenario 2: Run and Interpret Results
When user wants to run evals and understand scores:
- Call
waza_eval_runwith the eval spec path and context dir - Poll
waza_run_statusuntil complete (check every 10s) - Call
waza_results_summaryto get aggregate scores - Interpret the results for the user:
- Pass rate — percentage of tasks that passed all validators
- Weighted score — 0.0–1.0 aggregate across all tasks
- Duration — total and per-task execution time
- If pass rate < 80%, identify which tasks failed and why
- Call
waza_results_runsfor per-task details on failures - Suggest specific improvements: prompt rewording, validator tuning, fixture updates
Thresholds: ≥90% pass rate = strong, 70–89% = needs work, <70% = significant issues.
Scenario 3: Compare Models
When user wants to compare model performance:
- Ask which models to compare (e.g., gpt-4o vs claude-sonnet-4)
- Call
waza_eval_runwith model A — save results - Call
waza_eval_runwith model B — save results - Compare results side by side:
- Per-task pass/fail differences
- Score deltas (which model scores higher on which tasks)
- Duration differences (speed vs quality tradeoff)
- Provide a recommendation: which model is better for this skill and why
- Suggest next steps: try a third model, tune prompts for the weaker model, or adjust validators
Guidance: Run each model 2–3 times to account for variance before drawing conclusions.
Scenario 4: Debug a Failing Skill
When user's skill is failing evals or behaving unexpectedly:
- Call
waza_skill_checkto verify skill compliance (frontmatter, triggers, token count) - If compliance issues found, fix those first — they affect routing
- Call
waza_eval_runwith--verboseand--transcript-dirflags - Call
waza_results_runsto get per-task failure details - Analyze failure patterns:
- All tasks fail → prompt or fixture issue, check skill instructions
- Some tasks fail → specific edge cases, review failed task prompts
- Validator failures → regex too strict, code validator language mismatch
- Suggest targeted fixes based on the pattern
- Re-run with
waza_eval_runto verify the fix
Scenario 5: Ship Readiness Check
When user asks "is my skill ready?" or wants a pre-ship checklist:
- Call
waza_skill_check— verify compliance score ≥ medium-high - Call
waza_eval_validate— confirm eval YAML is valid - Call
waza_eval_run— execute full eval suite - Call
waza_results_summary— check aggregate scores - Render the readiness verdict:
SHIP READINESS CHECKLIST:
☐ Skill compliance: [score] (need: medium-high+)
☐ Eval YAML valid: [yes/no]
☐ Pass rate: [X]% (need: ≥90%)
☐ Weighted score: [X.XX] (need: ≥0.85)
☐ No task timeouts
☐ Consistent across 2+ runs
VERDICT: [READY / NOT READY — fix items marked ✗]
- If NOT READY, route to the appropriate scenario (Scenario 4 for failures, Scenario 1 for missing evals)
Conversation Style
- Always explain why before what — context before commands
- After every tool call, interpret the result in plain language
- When something fails, diagnose before suggesting fixes
- Offer the next logical step — don't wait to be asked
- Use the checklist format for multi-step validations
04 同仓库更多 Skills
waza-runner
Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.
waza
**WORKFLOW SKILL** - Evaluate AI agent skills using structured benchmarks with YAML specs, fixture isolation, and pluggable validators. USE FOR: run waza, waza help, run eval, run benchmark, evaluate skill, test agent, generate eval suite, init eval, compare results, score agent, agent evaluation, skill testing, cross-model comparison. DO NOT USE FOR: improving skill frontmatter (use waza dev), creating new skills from scratch (use skill-creator), token counting or budget checks (use waza tokens). INVOKES: Copilot SDK executor, mock engine, code/regex validators. FOR SINGLE OPERATIONS: use waza run directly for a single benchmark.
code-explainer
Explains code snippets in plain English, breaking down what the code does step by step. Perfect for learning, code reviews, or documentation.
code-explainer
**UTILITY SKILL** - Explain code snippets, functions, and algorithms in plain language. USE FOR: explain code, what does this code do, break down this function, help me understand this, walk through this algorithm, clarify this logic, explain this snippet, describe what happens here. DO NOT USE FOR: writing new code (use code generation), fixing bugs (use debugging), refactoring (use refactoring skills), code review with action items. INVOKES: file reading tools to access code, language detection for tailored explanations. FOR SINGLE OPERATIONS: If the user just needs to see file contents, use file reading tools directly.
profile-test-skill
**PROFILE TEST** - A skill for testing the profile command. USE FOR: testing profile analysis, verifying structural metrics. DO NOT USE FOR: production use.
count
暂无简介