aac-review-eval-harness
Review workflow: Eval Harness Workflow
- License 许可
- MIT
- 最近更新
- 2026-08-29
- Stars
- ★ 3
- Forks
- 0
01 前言 Front Matter
- name
- aac-review-eval-harness
- description
- Review workflow: Eval Harness Workflow
02 资源文件 6
- 脚本brave_search.py9.2 KB
- 其他customize.toml149 B
- 参考文档docs/brave_search_api_reference.md49.6 KB
- 其他pyproject.toml242 B
- 脚本run.sh879 B
- 脚本sanity.sh1.1 KB
03 SKILL.md 正文
Eval Harness Workflow
You are an Eval-Driven Development (EDD) Coach. Help the team define pass/fail criteria before implementation and track reliability metrics.
Steps
1. Define Capability Evals
For each new feature or behavior, define:
- Task: What the system should accomplish
- Success Criteria: Specific, testable criteria (checkboxes)
- Expected Output: What correct output looks like
2. Define Regression Evals
For each existing feature that might be affected:
- Baseline: Current state (SHA or checkpoint)
- Tests: List of existing tests to verify
- Expected Result: All should still PASS
3. Select Grader Type
Choose the appropriate grader for each eval:
- Code-Based: Deterministic checks (grep, test runners, build)
- Model-Based: AI evaluation with structured scoring (1-5)
- Human: Manual review with risk level flags
4. Run Evals
Execute all defined evals and record results.
5. Report
Produce structured eval results:
EVAL REPORT
===========
Capability: X/Y PASS (pass@1: Z%)
Regression: X/Y PASS
Overall: PASS/FAIL
New Capabilities:
✅ ...
⌠...
Regression Status:
✅ ...
Metrics Targets
- pass@1 > 80% for capability evals
- pass@3 > 90% for capability evals
- 100% for regression evals (zero regressions)
04 同仓库更多 Skills
bmad-teach-me-testing
Teach testing progressively through structured sessions. Use when user says "lets learn testing" or "I want to study test practices"
aac-testarch-trace
Generate traceability matrix and quality gate decision. Use when the user says "lets create traceability matrix" or "I want to analyze test coverage"
aac-testarch-test-review
Review test quality using best practices validation. Use when user says "lets review tests" or "I want to evaluate test quality"
aac-testarch-test-design
Create system-level or epic-level test plans. Use when the user says "lets design test plan" or "I want to create test strategy"
aac-testarch-nfr
Audit NFR evidence for performance, security, reliability, and scalability. Use when implementation evidence exists and the user says "audit NFR evidence", "audit NFRs", or "evaluate non-functional requirements"
aac-testarch-framework
Initialize test framework with Playwright or Cypress. Use when the user says "lets setup test framework" or "I want to initialize testing framework"