verify-bench
Vet a perf measurement (limiter, tuning, limits, errors, repeatability, relevance, did-it-happen) before reporting or acting. Use for benchmarks and speedup claims.
- License 许可
- MIT
- 最近更新
- 2026-11-05
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- verify-bench
- description
- Vet a perf measurement (limiter, tuning, limits, errors, repeatability, relevance, did-it-happen) before reporting or acting. Use for benchmarks and speedup claims.
02 资源文件 2
- 参考文档agents/openai.yaml89 B
- 参考文档CREDITS.md602 B
03 SKILL.md 正文
Benchmark checklist
Vet a perf measurement before reporting or acting on it: a PR before/after, a regression claim, a hillclimb harness, a library or config choice. Answer each question with run evidence, never a guess about code. perf-issue vets its baseline and every later number through this list; hillclimb vets its harness before freezing.
Ballpark rule: a quick estimate the user asked for takes one run, still checks errors and did-it-happen, says it is one run, and skips the rest unless the run looks wrong. A choice between options is never a ballpark.
Before running
Write the claim in shippable words ("export is 30% faster at p50 on the 60k-row dataset"); the questions test that sentence. Read the measurement script: what it times, counts, ignores. Check machine load and cores; a busy machine means interleaving both sides in the same noise and saying so.
The questions
- Why not double? Name the limiter from a profiling run (not the reported one), map it to source. Watch the load generator: saturated generator means you measured the generator. An unmoved number needs its limiter found before calling the change useless.
- Was it tuned? Every side runs as production does (builds, flags, env, pools, warm caches, versions, data). One side on defaults compares configurations, not implementations. Untunable means no winner from that run.
- Did it break limits? Do the arithmetic (bytes vs bandwidth, ops times cost vs cores, saved time vs piece time). Past a limit measured something else (cache, no-op, bug).
- Did it error? Count failures and wrong outputs, not just present ones. Fast rejections and slow retries both lie; add the count when the script lacks it.
- Does it reproduce? Each side at least 5 runs, alternating, median and range reported. Gaps under run-to-run variation are no measurable difference.
- Does it matter? Next to any micro result, measure the end-to-end waited path at realistic size and concurrency; report micro as a share of the whole.
- Did it even happen? Confirm the work ran inside the timed region (requests reached, rows written, bytes read, results used). Lazy or timed-out work produces numbers for nothing.
Report
Lead with the verdict (faster, slower, no measurable difference, inconclusive) plus number with unit, run count, range, and limiter. Inconclusive when the limiter is unnamed, a side ran untuned, or errors/happening went unchecked; name the gap. PR bodies carry one primary number; runs, range, and limiter live in a linked artifact.
04 同仓库更多 Skills
whips
Route any non-trivial task through one entry: match a playbook, drive HIT skills plus subagents, open a reviewed ready PR with ship-pr-shaped evidence.
verify-upkeep
Periodic upkeep for a project's verification skill: source wave plus live pass per feature, at most one PR of proven corrections. Use to audit the verify skill.
verify-statechart
Check a TLA+ model of a system that moves between states, such as a product flow, a protocol, or a service, and draw it as a Mermaid statechart. The report gives the shortest path to each state that breaks a rule. Use to show a bug as a path through states.
verify-new
Generate a project-local skill that drives the app like a user (launch, doctor, drive, evidence, feature map), proven once. Use when a project has no scripted proof.
ship-triage
Move issues and external PRs through a state machine of triage roles, categorise, verify, interview if needed, and write agent-ready briefs.
ship-runner
Run ready-for-agent issue queues on any schedule through /whips intake, one fresh worktree each, landing per risk tier, with the agent and model you pick at setup.