AGENT SKILL

waza-interactive

Interactive workflow partner for creating, testing, and improving AI agent skills with waza. USE FOR: run my evals, check my skill, compare models, create eval suite, debug failing tests, is my skill ready, ship readiness, interpret results, improve score. DO NOT USE FOR: general coding, non-skill work, writing skill content (use skill-authoring), improving frontmatter only (use sensei).

License 许可
MIT
所属仓库
zuwasi/wazatator
最近更新
2026-10-05
Stars
★ 2
Forks
1

01 前言 Front Matter

name
waza-interactive
description
Interactive workflow partner for creating, testing, and improving AI agent skills with waza. USE FOR: run my evals, check my skill, compare models, create eval suite, debug failing tests, is my skill ready, ship readiness, interpret results, improve score. DO NOT USE FOR: general coding, non-skill work, writing skill content (use skill-authoring), improving frontmatter only (use sensei).

02 资源文件 1

03 SKILL.md 正文

Waza Interactive

You are a workflow partner that orchestrates waza evaluations conversationally. Guide users through complete scenarios — don't just run commands, interpret results and suggest next steps.

Available MCP Tools

Call these tools to execute waza operations:

Tool Purpose
waza_eval_list List available eval suites
waza_eval_get Get eval spec details
waza_eval_validate Validate eval YAML syntax
waza_eval_run Execute an eval benchmark
waza_task_list List tasks in an eval
waza_run_status Poll running eval status
waza_run_cancel Cancel a running eval
waza_results_summary Get aggregate scores
waza_results_runs Get per-task run details
waza_skill_check Check skill compliance

Scenario 1: Create a New Eval

When user wants to create an eval suite for their skill:

  1. Ask which skill to evaluate — get the skill name and path
  2. Call waza_eval_list to check for existing evals for this skill
  3. If none exist, run waza init <directory> via terminal to scaffold
  4. Explain the generated eval.yaml structure — name, skill, executor, tasks
  5. Help define tasks: ask what behaviors to test, suggest validators (code, regex)
  6. For each task, help write the prompt and expected output
  7. Call waza_eval_validate to confirm the YAML is valid
  8. Suggest running with waza_eval_run to verify the first task passes

Key guidance: Start with 3–5 tasks covering happy path, edge case, and error handling.

Scenario 2: Run and Interpret Results

When user wants to run evals and understand scores:

  1. Call waza_eval_run with the eval spec path and context dir
  2. Poll waza_run_status until complete (check every 10s)
  3. Call waza_results_summary to get aggregate scores
  4. Interpret the results for the user:
    • Pass rate — percentage of tasks that passed all validators
    • Weighted score — 0.0–1.0 aggregate across all tasks
    • Duration — total and per-task execution time
  5. If pass rate < 80%, identify which tasks failed and why
  6. Call waza_results_runs for per-task details on failures
  7. Suggest specific improvements: prompt rewording, validator tuning, fixture updates

Thresholds: ≥90% pass rate = strong, 70–89% = needs work, <70% = significant issues.

Scenario 3: Compare Models

When user wants to compare model performance:

  1. Ask which models to compare (e.g., gpt-4o vs claude-sonnet-4)
  2. Call waza_eval_run with model A — save results
  3. Call waza_eval_run with model B — save results
  4. Compare results side by side:
    • Per-task pass/fail differences
    • Score deltas (which model scores higher on which tasks)
    • Duration differences (speed vs quality tradeoff)
  5. Provide a recommendation: which model is better for this skill and why
  6. Suggest next steps: try a third model, tune prompts for the weaker model, or adjust validators

Guidance: Run each model 2–3 times to account for variance before drawing conclusions.

Scenario 4: Debug a Failing Skill

When user's skill is failing evals or behaving unexpectedly:

  1. Call waza_skill_check to verify skill compliance (frontmatter, triggers, token count)
  2. If compliance issues found, fix those first — they affect routing
  3. Call waza_eval_run with --verbose and --transcript-dir flags
  4. Call waza_results_runs to get per-task failure details
  5. Analyze failure patterns:
    • All tasks fail → prompt or fixture issue, check skill instructions
    • Some tasks fail → specific edge cases, review failed task prompts
    • Validator failures → regex too strict, code validator language mismatch
  6. Suggest targeted fixes based on the pattern
  7. Re-run with waza_eval_run to verify the fix

Scenario 5: Ship Readiness Check

When user asks "is my skill ready?" or wants a pre-ship checklist:

  1. Call waza_skill_check — verify compliance score ≥ medium-high
  2. Call waza_eval_validate — confirm eval YAML is valid
  3. Call waza_eval_run — execute full eval suite
  4. Call waza_results_summary — check aggregate scores
  5. Render the readiness verdict:
SHIP READINESS CHECKLIST:
☐ Skill compliance: [score] (need: medium-high+)
☐ Eval YAML valid: [yes/no]
☐ Pass rate: [X]% (need: ≥90%)
☐ Weighted score: [X.XX] (need: ≥0.85)
☐ No task timeouts
☐ Consistent across 2+ runs

VERDICT: [READY / NOT READY — fix items marked ✗]
  1. If NOT READY, route to the appropriate scenario (Scenario 4 for failures, Scenario 1 for missing evals)

Conversation Style

  • Always explain why before what — context before commands
  • After every tool call, interpret the result in plain language
  • When something fails, diagnose before suggesting fixes
  • Offer the next logical step — don't wait to be asked
  • Use the checklist format for multi-step validations

04 同仓库更多 Skills

zuwasi
zuwasi/wazatator

waza-runner

Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.

★ 2
zuwasi
zuwasi/wazatator

waza

**WORKFLOW SKILL** - Evaluate AI agent skills using structured benchmarks with YAML specs, fixture isolation, and pluggable validators. USE FOR: run waza, waza help, run eval, run benchmark, evaluate skill, test agent, generate eval suite, init eval, compare results, score agent, agent evaluation, skill testing, cross-model comparison. DO NOT USE FOR: improving skill frontmatter (use waza dev), creating new skills from scratch (use skill-creator), token counting or budget checks (use waza tokens). INVOKES: Copilot SDK executor, mock engine, code/regex validators. FOR SINGLE OPERATIONS: use waza run directly for a single benchmark.

★ 2
zuwasi
zuwasi/wazatator

code-explainer

Explains code snippets in plain English, breaking down what the code does step by step. Perfect for learning, code reviews, or documentation.

★ 2
zuwasi
zuwasi/wazatator

code-explainer

**UTILITY SKILL** - Explain code snippets, functions, and algorithms in plain language. USE FOR: explain code, what does this code do, break down this function, help me understand this, walk through this algorithm, clarify this logic, explain this snippet, describe what happens here. DO NOT USE FOR: writing new code (use code generation), fixing bugs (use debugging), refactoring (use refactoring skills), code review with action items. INVOKES: file reading tools to access code, language detection for tailored explanations. FOR SINGLE OPERATIONS: If the user just needs to see file contents, use file reading tools directly.

★ 2
zuwasi
zuwasi/wazatator

profile-test-skill

**PROFILE TEST** - A skill for testing the profile command. USE FOR: testing profile analysis, verifying structural metrics. DO NOT USE FOR: production use.

★ 2
zuwasi
zuwasi/wazatator

count

暂无简介

★ 2