AGENT SKILL

claudea-bench

Compare `claudea` and normal `claude` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 6
Forks
0

01 前言 Front Matter

name
claudea-bench
description
Compare `claudea` and normal `claude` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.

03 SKILL.md 正文

Claudea benchmark

Use when asked to compare claudea with normal claude on a review, job, or task. By default, split comparable work evenly between the arms; --strict selects duplicate runs of the exact same task. Treat --strict as this skill's mode argument, not a flag for either CLI. The deliverable is a bounded, evidence-based comparison, including an explicit verdict. Do not benchmark this skill against itself or trigger another review workflow just to report the benchmark.

Set up a fair, bounded comparison

  1. Select a comparison mode before execution. By default, use split-work: freeze a bounded work set, shared acceptance rubric, and budget, then partition the work into distinct, non-overlapping units before either arm starts. Balance the halves as closely as practical by task type, estimated complexity, acceptance evidence, and input size; record units, estimates, assignments, and unavoidable imbalance. Keep constraints, safety limits, and per-arm budgets equivalent. Keep the planned partition and original scoring fixed. If delivery recovery requires reassignment or another runtime after an arm fails, record that work separately and exclude it from the original CLI arm's score. Do not share either arm's answers, findings, or intermediate reasoning. Record hashes for available work-set, input, rubric, and partition artifacts. When the caller supplies --strict, instead use strict same-task mode: freeze one exact task statement, input snapshot, rubric, and budget; give both arms identical task and input bytes and duplicate the task in separate isolated workspaces. Do not partition the strict task.
  2. For strict same-task mode, default to as-used unless the caller requests matched-model. Split-work always follows the matched-model rule in step 3. In either mode, invoke the actual shell claudea command once and the actual shell claude command once, in fresh isolated sessions. Preserve each command's normal configuration and harness; do not disable normal Claude instructions or features to make the arms look alike. Record the effective model, effort setting, config/profile identity, loaded instruction/skill context, working directory, executable or shell-function identity, and CLI version for each arm. Do not record secrets or credential values.
  3. For default split-work, or when the caller explicitly selects a matched-model strict comparison, select and set the same model and effort for both arms only for this task. Record the overrides and every remaining difference in configuration, instructions, skills, tools, wrappers, and working roots. Do not alter persistent defaults. Never infer that output differences came from cleanliness alone when other factors differ. Split-work is a matched-model comparison: use the same caller-selected task-scoped model and effort in both arms, and record any overrides. If either arm delegates to subagents because the caller requested it, keep each subagent within that arm's assigned work and isolated workspace; record its model, effort, and assignment. Never share workspaces or subagent outputs across arms. Record pre-existing unequal inputs or audits, and extra audits done during execution, separately from the planned split; do not count them as matched work or credit them to the assigned half.
  4. Use one bounded run per arm by default. Honor the caller's stated budget or the task owner's existing limits; do not invent extra spend or time. No automatic retry loop. If an arm fails or is unavailable, report the comparison as incomplete; the other arm is not the winner by default. If an arm reports an explicit usage limit, record that result and do not invoke the CLI again just to retest quota. Report any later work completed through another native route separately, identifying its runtime and excluding it from the original CLI arm's result. A retry requires an explicit, task-relevant reason and must remain within the authorized budget.

For a read-only review, give both arms the same frozen source snapshot and an explicit no-modifications constraint. If their configured execution modes cannot enforce read-only access, use separate disposable copies initialized from that snapshot and record this setup difference. For a mutating task, provide separate equivalent task-owned worktrees or fixtures, initialized from the same pinned state. Never let the arms write to the same checkout, shared production state, or each other's outputs. Do not repeat destructive, spending, credential, external-send, deployment, or other side effects: omit them from both arms or evaluate using safe isolated fixtures. The benchmark grants no authority beyond the original task.

Run the actual commands

Verify the locally available command shape before using it; shells may define claudea as a wrapper or function. The bundled shell helper uses native --safe-mode with Fable 5.2 (claude-fable-5-2) by default. Source ~/.claude/scripts/claudea.sh (or $CLAUDE_HOME/scripts/claudea.sh for a custom install) from the shell startup file after installation. It requires a Claude version with --safe-mode; verify claude --help first and fail visibly if unsupported. Do not silently substitute a model or remove isolation flags.

Safe mode disables user/project guidance, skills, custom agents, hooks, MCP servers, installed plugins, and auto memory. Authentication, admin policy, built-in tools/plugins, permissions, and normal-profile history remain available; this is configuration isolation, not a separate account or history store. The helper includes the same permission-bypass intent as codexa; use only within authorized task scope. It invokes the external CLI, bypassing a shell claude function. Record any resulting wrapper difference. Trailing caller flags can override the default model for a task. claudeac continues normal-profile history, which may include non-clean sessions; use fresh sessions for benchmarks.

For matched-model runs, use the installed Claude print interface in separate working directories and preserve structured receipts:

(cd "$WORKDIR_A" && claudea --print --output-format json --model "$MODEL" --effort "$EFFORT" < "$PROMPT_A") > "$OUTPUT_A"
(cd "$WORKDIR_B" && claude --print --output-format json --model "$MODEL" --effort "$EFFORT" < "$PROMPT_B") > "$OUTPUT_B"

For strict as-used runs, omit --model "$MODEL" --effort "$EFFORT" from both commands and retain each wrapper's configured defaults. For matched-model runs, use claude-fable-5-2 unless the caller selects another matched model. Verify actual response/model-usage metadata, not just the requested label. A model-access error leaves the comparison incomplete; preserve its exact sanitized error and never fall back automatically. Safe mode does not load /claudea-bench; orchestrate this skill from the normal agent session.

For split-work, keep shared instructions and rubric identical, include only each arm's assigned units and inputs, and record their hashes and sizes. For strict same-task, use identical prompt bytes and equivalent inputs. Choose separate workdirs according to the read-only or isolated-mutating setup above. If a local wrapper does not accept this shape, inspect its help/definition and adapt without replacing it with another model, provider, or configuration. Use the caller's/native timeout mechanism. Preserve private raw receipts in the approved scratch location; publish only sanitized material authorized for the task.

Evaluate and report

Where feasible, freeze both outputs (including hashes), label them A/B, and adjudicate before revealing which command produced which. Independently check each claimed issue against the pinned source and rubric. Deduplicate equivalent findings. Count valid unique findings by arm; record severity, false positives, missed planted or independently verified requirements, and task completion/correctness/safety. For split-work, report completion and correctness per assigned unit as well as per arm, and compare resource use only when measured on equivalent scopes. Treat task mix, input-size, audit, tool, and configuration differences as confounds; describe the observed outcomes without attributing them to a causal advantage from cleanliness or partitioning. Treat unknown recall as unknown, never as zero. More findings, longer answers, or more confident wording do not by themselves win. Correct, safe task completion takes priority over finding count.

Preserve native session transcripts when root/descendant usage verification is required; do not add --no-session-persistence if it removes those receipts. Capture root and descendant model/token/context data separately when exposed. Record the CLI version and field semantics; do not assume modelUsage separates or aggregates child usage without verification. Mark missing descendant usage or context explicitly unavailable, never zero. Keep transcripts private and publish only sanitized task-scoped extracts.

Compare elapsed time, tool use, output size, and token/cached-token data only when actually available; cached tokens are not dollars. Separate observed measurements from estimates. State confounds and what the result cannot establish.

End with exactly one verdict: claudea better, normal claude better, tie, or inconclusive. Name the supported dimension, each arm's valid-unique finding count, uncertainty, and the smallest next test only if another test is needed to resolve a material uncertainty. A single task supports a result for that task and setup, not a general model-quality claim. Write one durable Markdown report at the location required by the active environment's instructions; keep raw receipts private in approved scratch storage.

04 同仓库更多 Skills

jleechanorg
jleechanorg/jleechan-skills

disk-reclaim-playbook

Use Disk Magician to diagnose disk growth or qualify cleanup on macOS or Linux.

★ 6
jleechanorg
jleechanorg/jleechan-skills

pr

Alias for the canonical pr-lifecycle PR workflow skill (/pr-workflow, /prw).

★ 6
jleechanorg
jleechanorg/jleechan-skills

whisper

OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.

★ 6
jleechanorg
jleechanorg/jleechan-skills

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

★ 6
jleechanorg
jleechanorg/jleechan-skills

pr-report

This skill should be used when the user asks for a PR report, PR audit, PR delta analysis, /pr-report, or wants per-PR summary with delta files/lines. Generates a structured report for one or more open PRs covering purpose, files changed, +/− line counts, /code-standards + /thermo review findings, /ponytail simplification opportunities, file-overlap combination candidates, and /green status per pr-green-definition. Output: machine-readable JSONL + human-readable markdown. Read-only by default; can apply simplifications if user opts in.

★ 6
jleechanorg
jleechanorg/jleechan-skills

pr-description-format

Required PR-description format pointer; use the shared owner and target repository template.

★ 6