AGENT SKILL

pair-benchmark-all-executors

Run one benchmark script that compares pairv2, pair via Claude Teams, and pair via direct Python

License 许可
MIT
最近更新
2026-11-14
Stars
★ 6
Forks
0

01 前言 Front Matter

name
pair-benchmark-all-executors
type
usage
scope
project
description
Run one benchmark script that compares pairv2, pair via Claude Teams, and pair via direct Python

03 SKILL.md 正文

Pair Benchmark (All Executors)

Purpose

Use one script to benchmark all pair executors:

  • pairv2 (.claude/pair/pair_execute_v2.py)
  • /pair via Claude Teams prompt path
  • /pair via direct Python (.claude/pair/pair_execute.py)

Script

python .claude/pair/benchmark_pair_executors.py (or venv/bin/python when using a virtualenv)

Quick Start

Run all three executors in one benchmark:

python .claude/pair/benchmark_pair_executors.py \
  --executor-set all \
  --benchmark-iterations 1 \
  --timeout-seconds 180 \
  --teams-command "claude --print" \
  --artifact /tmp/{repo}/{branch}/pair/all_three.json

Run a small, named Amazon-style ecommerce preset:

python .claude/pair/benchmark_pair_executors.py \
  --task-preset amazon_clone \
  --pairv2-max-cycles 2 \
  --timeout-seconds 1200 \
  --artifact /tmp/{repo}/{branch}/pair/amazon_clone_benchmark.json

For historical benchmark prompts, including alternate Amazon-style variants, see testing_llm/pair/benchmark_tasks.json.

Run all three in parallel:

python .claude/pair/benchmark_pair_executors.py \
  --executor-set all \
  --parallel \
  --benchmark-iterations 1 \
  --timeout-seconds 180 \
  --teams-command "claude --print" \
  --artifact /tmp/{repo}/{branch}/pair/all_three_parallel.json

Executor Modes

  • --executor-set all: pair python + pair teams + pairv2
  • --executor-set legacy-vs-v2: pair python vs pairv2
  • --executor-set teams-vs-v2: pair teams vs pairv2
  • --executor-set pairv2-only: pairv2 only

Key Output Fields

  • pair_python, pair_python_status_counts, pair_python_raw
  • pair_claude_teams, pair_teams_status_counts, pair_teams_raw
  • pairv2_langgraph, pairv2_status_counts, pairv2_raw
  • speedup_ratio_pair_python_over_v2
  • speedup_ratio_pair_teams_over_v2
  • speedup_ratio_pair_python_over_teams

Current Defaults (updated Feb 2026)

  • --executor-set: pairv2-only
  • --coder-cli: minimax
  • --verifier-cli: minimax
  • --task: hello world script + pytest test (default); overridden by --task-preset when provided
  • --timeout-seconds: 600

Run with no args to use these defaults:

venv/bin/python .claude/pair/benchmark_pair_executors.py

Notes

  • minimax is not a CLI binary — it runs claude with MiniMax API endpoint.

04 同仓库更多 Skills

jleechanorg
jleechanorg/jleechan-skills

disk-reclaim-playbook

Use Disk Magician to diagnose disk growth or qualify cleanup on macOS or Linux.

★ 6
jleechanorg
jleechanorg/jleechan-skills

claudea-bench

Compare `claudea` and normal `claude` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.

★ 6
jleechanorg
jleechanorg/jleechan-skills

pr

Alias for the canonical pr-lifecycle PR workflow skill (/pr-workflow, /prw).

★ 6
jleechanorg
jleechanorg/jleechan-skills

whisper

OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.

★ 6
jleechanorg
jleechanorg/jleechan-skills

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

★ 6
jleechanorg
jleechanorg/jleechan-skills

pr-report

This skill should be used when the user asks for a PR report, PR audit, PR delta analysis, /pr-report, or wants per-PR summary with delta files/lines. Generates a structured report for one or more open PRs covering purpose, files changed, +/− line counts, /code-standards + /thermo review findings, /ponytail simplification opportunities, file-overlap combination candidates, and /green status per pr-green-definition. Output: machine-readable JSONL + human-readable markdown. Read-only by default; can apply simplifications if user opts in.

★ 6