[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7E8DAQAAKWGJP2BKAQMVBQ1":3,"skill-same:01M7E8DAQAAKWGJP2BKAQMVBQ1":61},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":14,"repositoryHtmlUrl":37,"repositoryHomepageUrl":6,"ownerHtmlUrl":38,"ownerBlogUrl":39,"ownerDescription":40,"updatedAt":41,"install":42,"resources":54},"01M7E8DAQAAKWGJP2BKAQMVBQ1","01-benchmark",null,"Use when benchmarking pytest optimization hypotheses from 00-scan in isolation to identify speedups that beat measured noise.","tony\u002Fskills","tony","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F26336?v=4","01M72E5RNJ7SQV8F9KBM5RKRMN","Tony Narlock","01M7E7MBN5C6YZ4JJJ5Y82KBJG",0,"MIT","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fpytest-optimizer\u002Fskills\u002F01-benchmark\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: 01-benchmark\ndescription: \"Use when benchmarking pytest optimization hypotheses from 00-scan in isolation to identify speedups that beat measured noise.\"\nallowed-tools: [\"Bash\", \"Read\", \"Grep\", \"Glob\", \"Write\", \"Edit\", \"AskUserQuestion\"]\nargument-hint: \"[--only=\u003Cheuristic-id>] [--runs=N] [--force] [--memory-dir=\u003Cpath>]\"\ndisable-model-invocation: true\n---\n\n# 01-benchmark\n\nProve or disprove each hypothesis by measurement, applying candidates only\ninside an owned scratch worktree, never on the user's working tree or history,\nand keep each validated change as a hash-bound patch for `03-execute`.\n\n`$ARGUMENTS` may pass `--only=\u003Cheuristic-id>` to benchmark a subset, `--runs=N`\n(measurement runs per candidate, default matches the baseline), `--force` to\nre-benchmark ids already recorded, and `--memory-dir` to override the memory\nlocation.\n\nArithmetic goes through `..\u002F..\u002Fscripts\u002Fmeasure.py` and scratch worktrees and\npatches through `..\u002F..\u002Fscripts\u002Fisolate.py` (Python 3 standard library). Both\npaths resolve from this skill's own directory, not the project's working\ndirectory: resolve this file's directory, join each path, print the absolute\npaths once, and write those literal paths in each command below where\n`\u003Cmeasure.py>` and `\u003Cisolate.py>` appear.\n\n## Step 1: Load inputs\n\nResolve the memory directory by the lookup order in\n`..\u002F..\u002Freferences\u002Fmemory-schema.md`. If none exists, or `baseline.json` or\n`hypotheses.json` is missing, stop with the halted-run shape in\n`..\u002F..\u002Freferences\u002Foutput-contract.md` and tell the user to run\n`pytest-optimizer:00-scan` first.\n\nCheck that the baseline still describes this checkout. The command names the\nfirst failure only (no token, a dirty scan, another `HEAD`, a working tree whose\ndigest differs from the scan's, another test command or runner `pytest\n--version`):\n\n```console\n$ python3 \u003Cisolate.py> stale \\\n    --memory-dir \u003Cmemory-dir> \\\n    --phase benchmark \\\n    --command \"\u003Ctest_command>\" \\\n    --pytest-version \u003Cversion>\n```\n\nExit 1 means the baseline is stale or came from a dirty tree: print its `reason`\nand tell the user to commit or shelve their changes and run\n`pytest-optimizer:00-scan --force`. The digest check also covers uncommitted\nedits and untracked files, which the scratch worktree, cut from `HEAD`, would\nlack.\n\nRead `baseline.json`. Substitute `test_command` from `baseline.json` wherever this\nfile writes `pytest`. List the hypotheses to benchmark; the command skips an id\nwhose `benchmarks.json` entry already carries `state.json`'s `env_token`, unless\n`--force`, and honors `--only` (a hypothesis id or a heuristic id):\n\n```console\n$ python3 \u003Cmeasure.py> pending \\\n    --memory-dir \u003Cmemory-dir>\n```\n\nResolve `\u003Cruns-dir>` as `..\u002F..\u002Freferences\u002Fmemory-schema.md` (\"Run files\") says\nand create it. Delete each run file before its first timed run.\n\n## Step 2: Budget, control run, then each hypothesis\n\n**Budget.** Take `t` from one run of `test_command` (the baseline median\nworks) and price the phase: the control plus every open speed hypothesis. The\nproject's own limit, when its docs set one, replaces the 1800-second default:\n\n```console\n$ python3 \u003Cmeasure.py> budget \\\n    --phase benchmark \\\n    --seconds \u003Ct> \\\n    --runs \u003CN> \\\n    --hypotheses \u003Copen speed hypotheses>\n```\n\nWhen `over` is true, ask the user to confirm, narrow with `--only`, or lower\n`--runs`. Without an interactive question tool, benchmark only the `fits`\nhypotheses, lowest prior risk and effort first, leave the rest `open`, and say\nso in the closing line; when `control_fits` is false, stop and print the\nestimate.\n\n**Control run.** The baseline ran in the user's tree; candidates run in a fresh\nworktree with no bytecode cache and a new virtualenv, so the two differ before\nany change. Run per-hypothesis steps 1, 3 and 5 once with no change applied: create the\nworktree, time the measurement command there (skip step 3's `compare`, since no\ncontrol exists yet), append its runs to `\u003Cruns-dir>\u002Fcontrol-runs.jsonl`, and\ndiscard the worktree. Save its summary in the memory directory:\n\n```console\n$ python3 \u003Cmeasure.py> band \u003Cruns-dir>\u002Fcontrol-runs.jsonl > \u003Cmemory-dir>\u002Fcontrol.json\n```\n\nWhen `band` exits 1 the control run is invalid (non-zero exit, too few runs,\nunavailable or changed test counts): stop, report its `reason` in one sentence, and run no candidate.\n\nThen prove the fresh checkout runs the scan's tests. Read `valid` only from this\ncomparison, since the control is not a candidate:\n\n```console\n$ python3 \u003Cmeasure.py> compare \\\n    \u003Cmemory-dir>\u002Fbaseline.json \\\n    \u003Cruns-dir>\u002Fcontrol-runs.jsonl\n```\n\nWrite the control's listing from `test_command` run in `workdir`, as `00-scan`\nStep 2 does, to `\u003Cruns-dir>\u002Fcontrol-collection.txt`, and compare node ids:\n\n```console\n$ python3 \u003Cmeasure.py> ids \\\n    \u003Cmemory-dir>\u002Fcollection.txt \\\n    \u003Cruns-dir>\u002Fcontrol-collection.txt\n```\n\nWhen either exits 1 the fresh checkout is not the suite the scan measured (tests\nmissing from the checkout, an environment that was not rebuilt): stop, report\nthe `reason` in one sentence, and run no candidate.\n\nJudge every candidate against `control.json`, not `baseline.json`. When the\ncontrol's median differs from the baseline's by more than the baseline band,\nsay so in the closing line: absolute times from this phase are not comparable\nto the baseline's. When it does not differ, say nothing about the control.\n\n**Per hypothesis.** For each open hypothesis, in isolation:\n\n1. **Isolate.** Create an owned scratch worktree at the baseline commit. The\n   command refuses a repository root that is or contains the home directory, and\n   prints `dir`, `tree` and `workdir`, so use those literal paths in every later\n   command:\n\n   ```console\n   $ python3 \u003Cisolate.py> start \\\n       --repo . \\\n       --base \u003Cbase_head>\n   ```\n\n   Run everything below inside `workdir` (the output's path for the directory the\n   skill was started from: `tree` itself, or the project's subdirectory when the\n   repository holds several projects) with the project's own runner. The worktree\n   holds only tracked files: no virtualenv and no gitignored build artifacts.\n   Rebuild them there with the project's install or sync command (`uv run`\n   creates a missing project environment on first use, per the [uv project\n   layout](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fconcepts\u002Fprojects\u002Flayout\u002F) docs, which the\n   untimed warm-up absorbs), or mark the hypothesis `rejected` when the suite\n   cannot run from a clean checkout.\n\n   When `start` fails because `git worktree add` fails (its `reason` quotes git),\n   add `--export` to the command: the scratch directory then holds a copy of\n   `base_head` in its own one-commit repository, and every later command works\n   the same.\n\n   An editable install loads the project folder it was installed from\n   ([setuptools development mode](https:\u002F\u002Fsetuptools.pypa.io\u002Fen\u002Flatest\u002Fuserguide\u002Fdevelopment_mode.html)).\n   When the runner reuses the user's environment, a hypothesis that edits package\n   source would be measured against the original tree. Check where the package\n   imports from, under the runner and from `workdir`, before measuring such a\n   hypothesis; if the printed path is outside `tree`, mark it `rejected` with that\n   reason (`\u003Crunner>` is the project's runner prefix, such as `uv run`):\n\n   ```console\n   $ \u003Crunner> python -c \"import \u003Cpackage>; print(\u003Cpackage>.__file__)\"\n   ```\n2. **Apply** the single change the heuristic prescribes\n   (`..\u002F..\u002Freferences\u002Fheuristic-catalog.md`), then capture it before running\n   anything, since every file a run creates would enter the patch:\n\n   ```console\n   $ python3 \u003Cisolate.py> capture \\\n       --cwd \u003Ctree> \\\n       --out \u003Cmemory-dir>\u002Fpatches\u002F\u003Chypothesis-id>.patch \\\n       > \u003Cruns-dir>\u002F\u003Chypothesis-id>-capture.json\n   ```\n\n   Create `patches\u002F` first. The output holds the patch's `sha256` and `paths`;\n   `03-execute` applies exactly these bytes.\n3. **Measure.** Run the measurement command\n   (`..\u002F..\u002Freferences\u002Fdurations-parsing.md`) once untimed as a warm-up and then\n   `--runs` times, appending each JSON line to\n   `\u003Cruns-dir>\u002F\u003Chypothesis-id>-runs.jsonl`:\n\n   ```console\n   $ python3 \u003Cmeasure.py> run \\\n       --cwd \u003Cworkdir> \\\n       -- pytest -p no:cacheprovider -p no:randomly \\\n       >> \u003Cruns-dir>\u002F\u003Chypothesis-id>-runs.jsonl\n   ```\n\n   Judge the runs against the control:\n\n   ```console\n   $ python3 \u003Cmeasure.py> compare \u003Cmemory-dir>\u002Fcontrol.json \u003Cruns-dir>\u002F\u003Chypothesis-id>-runs.jsonl\n   ```\n\n   When `valid`, write the listing from `test_command` run in `workdir`, as\n   `00-scan` Step 2 does, and compare node ids with the scan's:\n\n   ```console\n   $ python3 \u003Cmeasure.py> ids \\\n       \u003Cmemory-dir>\u002Fcollection.txt \\\n       \u003Cruns-dir>\u002F\u003Chypothesis-id>-collection.txt \\\n       > \u003Cruns-dir>\u002F\u003Chypothesis-id>-ids.json\n   ```\n\n   A candidate counts only when `valid` is true, `clears_noise` is true, and the\n   ids are `same`, unless it is a deliberate deselection or hygiene change\n   below. Equal counts alone would pass a swapped test.\n   Otherwise `impact = 0` and it is rejected with the reported reason. A count\n   drop is a rejection naming the tests lost; it never counts as a speedup unless\n   the change is a deliberate deselection below.\n\n   A cross-run cache candidate measures with the cache provider enabled:\n   drop `-p no:cacheprovider` for it, so the untimed warm-up populates the cache\n   and the timed runs hit it. Record `cache: \"enabled\"`; `03-execute` verifies\n   and re-measures it the same way. Its saving applies from the second run\n   onward; say so in the record.\n\n   A deselecting candidate wins by running fewer tests, so it passes\n   `--expect-total \u003Ccount>`, records `removes_tests` with the number dropped and\n   `removed_ids` with the node ids the `ids` command lists as `missing`, and\n   reports the saving as time not spent on those tests, not as a faster suite.\n   `02-plan` shows the count at the approval gate.\n\n   This permits removal of passing outcomes only: the baseline's non-passing\n   count must remain unchanged. Removing skipped or xfailed cases needs a\n   separate selection policy; an expected total does not authorize it.\n\n   A typing or parametrize hypothesis saves no time by design: skip the\n   timing. Run the suite once and the project's type checker over the test\n   directory; it counts when both pass and the test count is unchanged (node ids\n   may change by design). Record it\n   with `kind: \"hygiene\"` and `median_delta` and `clears_noise` null.\n4. **Safety gates** per `..\u002F..\u002Freferences\u002Fsafety-gates.md`: the `valid` check\n   above for every change; order independence and collection determinism\n   for a change touching order, scope, or parallelism. A gate failure\n   rejects the candidate and surfaces the prerequisite refactor as its own\n   follow-up hypothesis. A parallelism candidate also runs 5-10 times, with the\n   grouping and single-exec fixes from the same reference. It measures the control command (serial, as in\n   step 3) and the same command plus `-n auto`, both recorded, the second as\n   parallel; the no-xdist rule covers every other candidate.\n5. **Discard** this hypothesis's scratch directory. The command removes only the\n   directory step 1 created and leaves every other worktree and branch alone:\n\n   ```console\n   $ python3 \u003Cisolate.py> finish \\\n       --dir \u003Cdir>\n   ```\n\n   Confirm `finish` printed `removed` true, and that `isolate.py inputs --cwd .`\n   still prints the `inputs_digest` in `state.json`; otherwise stop and report\n   the difference.\n\n## Step 3: Record\n\nUpsert each benchmarked id's entry in the shape `..\u002F..\u002Freferences\u002Fmemory-schema.md`\ngives. Save the `capture` output (step 2) and the `ids` output (step 3) as files\nin `\u003Cruns-dir>`; the command reads the patch path, hash and files from the first\nand, for a deselecting candidate, `removed_ids` and `removes_tests` from the\nsecond. Add the measured fields with `--set`:\n\n```console\n$ python3 \u003Cmeasure.py> record \u003Cmemory-dir>\u002Fbenchmarks.json \\\n    --state \u003Cmemory-dir>\u002Fstate.json \\\n    --id \u003Chypothesis-id> \\\n    --capture-json \u003Cruns-dir>\u002F\u003Chypothesis-id>-capture.json \\\n    --ids-json \u003Cruns-dir>\u002F\u003Chypothesis-id>-ids.json \\\n    --set 'median_delta=\u003Cseconds>' \\\n    --set 'clears_noise=\u003Ctrue|false>' \\\n    --set 'verdict=\"\u003Cvalidated|rejected>\"'\n```\n\nThe command stamps `env_token` and `base_head`, keeps the entries for ids this\nrun skipped (`--only` or a matching token) and replaces those for ids it\nbenchmarked, `--force` included. Set `confidence` high only for serial, repeated runs with the delta above the\nfloor. Nothing in this phase touches the user's tree, history, or any remote. Then mark the\nphase done, which clears the later flags:\n\n```console\n$ python3 \u003Cmeasure.py> state \\\n    complete \u003Cmemory-dir>\u002Fstate.json \\\n    --phase benchmark\n```\n\n## Step 4: Report\n\nEmit the `01-benchmark` sections from `..\u002F..\u002Freferences\u002Foutput-contract.md`. Write\nthe prose per `..\u002F..\u002Freferences\u002Fprose-rules.md`. Put each candidate under the\nsection its measurement earned: a candidate rejected for a count drop or a gate\nfailure is under `## Rejected`, never `## Validated`. An empty section says\n\"None.\" in one line. Advise nothing the runs did not measure (no \"make X\ncheaper\"). When the phase halted before any run, print the halted-run shape\ninstead, with one cause and no list of what is absent. Reply in this shape, as\nplain text outside any code fence, with plain bullets, no bold (never\n`- **Label:** ...`), and no local paths:\n\n```text\n✓ 1 validated, 1 rejected\n\n## Validated\n\n- H10 mis-scoped fixture: db_session to session scope saves 4.6s, above the 0.44s band, confidence 0.9.\n\n## Rejected\n\n- H33 fixed sleep in test_report: removing it saves 0.02s, inside the noise band.\n\n## Safety-gate results\n\n- H10 mis-scoped fixture: order independence held across 3 seeds and collection is deterministic.\n```\n\nClose with an `AskUserQuestion` panel offering to build the plan, re-benchmark a\nsubset, or stop. Without one, follow the contract's next-step rule.\n","# 01-benchmark\n\nProve or disprove each hypothesis by measurement, applying candidates only\ninside an owned scratch worktree, never on the user's working tree or history,\nand keep each validated change as a hash-bound patch for `03-execute`.\n\n`$ARGUMENTS` may pass `--only=\u003Cheuristic-id>` to benchmark a subset, `--runs=N`\n(measurement runs per candidate, default matches the baseline), `--force` to\nre-benchmark ids already recorded, and `--memory-dir` to override the memory\nlocation.\n\nArithmetic goes through `..\u002F..\u002Fscripts\u002Fmeasure.py` and scratch worktrees and\npatches through `..\u002F..\u002Fscripts\u002Fisolate.py` (Python 3 standard library). Both\npaths resolve from this skill's own directory, not the project's working\ndirectory: resolve this file's directory, join each path, print the absolute\npaths once, and write those literal paths in each command below where\n`\u003Cmeasure.py>` and `\u003Cisolate.py>` appear.\n\n## Step 1: Load inputs\n\nResolve the memory directory by the lookup order in\n`..\u002F..\u002Freferences\u002Fmemory-schema.md`. If none exists, or `baseline.json` or\n`hypotheses.json` is missing, stop with the halted-run shape in\n`..\u002F..\u002Freferences\u002Foutput-contract.md` and tell the user to run\n`pytest-optimizer:00-scan` first.\n\nCheck that the baseline still describes this checkout. The command names the\nfirst failure only (no token, a dirty scan, another `HEAD`, a working tree whose\ndigest differs from the scan's, another test command or runner `pytest\n--version`):\n\n```console\n$ python3 \u003Cisolate.py> stale \\\n    --memory-dir \u003Cmemory-dir> \\\n    --phase benchmark \\\n    --command \"\u003Ctest_command>\" \\\n    --pytest-version \u003Cversion>\n```\n\nExit 1 means the baseline is stale or came from a dirty tree: print its `reason`\nand tell the user to commit or shelve their changes and run\n`pytest-optimizer:00-scan --force`. The digest check also covers uncommitted\nedits and untracked files, which the scratch worktree, cut from `HEAD`, would\nlack.\n\nRead `baseline.json`. Substitute `test_command` from `baseline.json` wherever this\nfile writes `pytest`. List the hypotheses to benchmark; the command skips an id\nwhose `benchmarks.json` entry already carries `state.json`'s `env_token`, unless\n`--force`, and honors `--only` (a hypothesis id or a heuristic id):\n\n```console\n$ python3 \u003Cmeasure.py> pending \\\n    --memory-dir \u003Cmemory-dir>\n```\n\nResolve `\u003Cruns-dir>` as `..\u002F..\u002Freferences\u002Fmemory-schema.md` (\"Run files\") says\nand create it. Delete each run file before its first timed run.\n\n## Step 2: Budget, control run, then each hypothesis\n\n**Budget.** Take `t` from one run of `test_command` (the baseline median\nworks) and price the phase: the control plus every open speed hypothesis. The\nproject's own limit, when its docs set one, replaces the 1800-second default:\n\n```console\n$ python3 \u003Cmeasure.py> budget \\\n    --phase benchmark \\\n    --seconds \u003Ct> \\\n    --runs \u003CN> \\\n    --hypotheses \u003Copen speed hypotheses>\n```\n\nWhen `over` is true, ask the user to confirm, narrow with `--only`, or lower\n`--runs`. Without an interactive question tool, benchmark only the `fits`\nhypotheses, lowest prior risk and effort first, leave the rest `open`, and say\nso in the closing line; when `control_fits` is false, stop and print the\nestimate.\n\n**Control run.** The baseline ran in the user's tree; candidates run in a fresh\nworktree with no bytecode cache and a new virtualenv, so the two differ before\nany change. Run per-hypothesis steps 1, 3 and 5 once with no change applied: create the\nworktree, time the measurement command there (skip step 3's `compare`, since no\ncontrol exists yet), append its runs to `\u003Cruns-dir>\u002Fcontrol-runs.jsonl`, and\ndiscard the worktree. Save its summary in the memory directory:\n\n```console\n$ python3 \u003Cmeasure.py> band \u003Cruns-dir>\u002Fcontrol-runs.jsonl > \u003Cmemory-dir>\u002Fcontrol.json\n```\n\nWhen `band` exits 1 the control run is invalid (non-zero exit, too few runs,\nunavailable or changed test counts): stop, report its `reason` in one sentence, and run no candidate.\n\nThen prove the fresh checkout runs the scan's tests. Read `valid` only from this\ncomparison, since the control is not a candidate:\n\n```console\n$ python3 \u003Cmeasure.py> compare \\\n    \u003Cmemory-dir>\u002Fbaseline.json \\\n    \u003Cruns-dir>\u002Fcontrol-runs.jsonl\n```\n\nWrite the control's listing from `test_command` run in `workdir`, as `00-scan`\nStep 2 does, to `\u003Cruns-dir>\u002Fcontrol-collection.txt`, and compare node ids:\n\n```console\n$ python3 \u003Cmeasure.py> ids \\\n    \u003Cmemory-dir>\u002Fcollection.txt \\\n    \u003Cruns-dir>\u002Fcontrol-collection.txt\n```\n\nWhen either exits 1 the fresh checkout is not the suite the scan measured (tests\nmissing from the checkout, an environment that was not rebuilt): stop, report\nthe `reason` in one sentence, and run no candidate.\n\nJudge every candidate against `control.json`, not `baseline.json`. When the\ncontrol's median differs from the baseline's by more than the baseline band,\nsay so in the closing line: absolute times from this phase are not comparable\nto the baseline's. When it does not differ, say nothing about the control.\n\n**Per hypothesis.** For each open hypothesis, in isolation:\n\n1. **Isolate.** Create an owned scratch worktree at the baseline commit. The\n   command refuses a repository root that is or contains the home directory, and\n   prints `dir`, `tree` and `workdir`, so use those literal paths in every later\n   command:\n\n   ```console\n   $ python3 \u003Cisolate.py> start \\\n       --repo . \\\n       --base \u003Cbase_head>\n   ```\n\n   Run everything below inside `workdir` (the output's path for the directory the\n   skill was started from: `tree` itself, or the project's subdirectory when the\n   repository holds several projects) with the project's own runner. The worktree\n   holds only tracked files: no virtualenv and no gitignored build artifacts.\n   Rebuild them there with the project's install or sync command (`uv run`\n   creates a missing project environment on first use, per the [uv project\n   layout](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fconcepts\u002Fprojects\u002Flayout\u002F) docs, which the\n   untimed warm-up absorbs), or mark the hypothesis `rejected` when the suite\n   cannot run from a clean checkout.\n\n   When `start` fails because `git worktree add` fails (its `reason` quotes git),\n   add `--export` to the command: the scratch directory then holds a copy of\n   `base_head` in its own one-commit repository, and every later command works\n   the same.\n\n   An editable install loads the project folder it was installed from\n   ([setuptools development mode](https:\u002F\u002Fsetuptools.pypa.io\u002Fen\u002Flatest\u002Fuserguide\u002Fdevelopment_mode.html)).\n   When the runner reuses the user's environment, a hypothesis that edits package\n   source would be measured against the original tree. Check where the package\n   imports from, under the runner and from `workdir`, before measuring such a\n   hypothesis; if the printed path is outside `tree`, mark it `rejected` with that\n   reason (`\u003Crunner>` is the project's runner prefix, such as `uv run`):\n\n   ```console\n   $ \u003Crunner> python -c \"import \u003Cpackage>; print(\u003Cpackage>.__file__)\"\n   ```\n2. **Apply** the single change the heuristic prescribes\n   (`..\u002F..\u002Freferences\u002Fheuristic-catalog.md`), then capture it before running\n   anything, since every file a run creates would enter the patch:\n\n   ```console\n   $ python3 \u003Cisolate.py> capture \\\n       --cwd \u003Ctree> \\\n       --out \u003Cmemory-dir>\u002Fpatches\u002F\u003Chypothesis-id>.patch \\\n       > \u003Cruns-dir>\u002F\u003Chypothesis-id>-capture.json\n   ```\n\n   Create `patches\u002F` first. The output holds the patch's `sha256` and `paths`;\n   `03-execute` applies exactly these bytes.\n3. **Measure.** Run the measurement command\n   (`..\u002F..\u002Freferences\u002Fdurations-parsing.md`) once untimed as a warm-up and then\n   `--runs` times, appending each JSON line to\n   `\u003Cruns-dir>\u002F\u003Chypothesis-id>-runs.jsonl`:\n\n   ```console\n   $ python3 \u003Cmeasure.py> run \\\n       --cwd \u003Cworkdir> \\\n       -- pytest -p no:cacheprovider -p no:randomly \\\n       >> \u003Cruns-dir>\u002F\u003Chypothesis-id>-runs.jsonl\n   ```\n\n   Judge the runs against the control:\n\n   ```console\n   $ python3 \u003Cmeasure.py> compare \u003Cmemory-dir>\u002Fcontrol.json \u003Cruns-dir>\u002F\u003Chypothesis-id>-runs.jsonl\n   ```\n\n   When `valid`, write the listing from `test_command` run in `workdir`, as\n   `00-scan` Step 2 does, and compare node ids with the scan's:\n\n   ```console\n   $ python3 \u003Cmeasure.py> ids \\\n       \u003Cmemory-dir>\u002Fcollection.txt \\\n       \u003Cruns-dir>\u002F\u003Chypothesis-id>-collection.txt \\\n       > \u003Cruns-dir>\u002F\u003Chypothesis-id>-ids.json\n   ```\n\n   A candidate counts only when `valid` is true, `clears_noise` is true, and the\n   ids are `same`, unless it is a deliberate deselection or hygiene change\n   below. Equal counts alone would pass a swapped test.\n   Otherwise `impact = 0` and it is rejected with the reported reason. A count\n   drop is a rejection naming the tests lost; it never counts as a speedup unless\n   the change is a deliberate deselection below.\n\n   A cross-run cache candidate measures with the cache provider enabled:\n   drop `-p no:cacheprovider` for it, so the untimed warm-up populates the cache\n   and the timed runs hit it. Record `cache: \"enabled\"`; `03-execute` verifies\n   and re-measures it the same way. Its saving applies from the second run\n   onward; say so in the record.\n\n   A deselecting candidate wins by running fewer tests, so it passes\n   `--expect-total \u003Ccount>`, records `removes_tests` with the number dropped and\n   `removed_ids` with the node ids the `ids` command lists as `missing`, and\n   reports the saving as time not spent on those tests, not as a faster suite.\n   `02-plan` shows the count at the approval gate.\n\n   This permits removal of passing outcomes only: the baseline's non-passing\n   count must remain unchanged. Removing skipped or xfailed cases needs a\n   separate selection policy; an expected total does not authorize it.\n\n   A typing or parametrize hypothesis saves no time by design: skip the\n   timing. Run the suite once and the project's type checker over the test\n   directory; it counts when both pass and the test count is unchanged (node ids\n   may change by design). Record it\n   with `kind: \"hygiene\"` and `median_delta` and `clears_noise` null.\n4. **Safety gates** per `..\u002F..\u002Freferences\u002Fsafety-gates.md`: the `valid` check\n   above for every change; order independence and collection determinism\n   for a change touching order, scope, or parallelism. A gate failure\n   rejects the candidate and surfaces the prerequisite refactor as its own\n   follow-up hypothesis. A parallelism candidate also runs 5-10 times, with the\n   grouping and single-exec fixes from the same reference. It measures the control command (serial, as in\n   step 3) and the same command plus `-n auto`, both recorded, the second as\n   parallel; the no-xdist rule covers every other candidate.\n5. **Discard** this hypothesis's scratch directory. The command removes only the\n   directory step 1 created and leaves every other worktree and branch alone:\n\n   ```console\n   $ python3 \u003Cisolate.py> finish \\\n       --dir \u003Cdir>\n   ```\n\n   Confirm `finish` printed `removed` true, and that `isolate.py inputs --cwd .`\n   still prints the `inputs_digest` in `state.json`; otherwise stop and report\n   the difference.\n\n## Step 3: Record\n\nUpsert each benchmarked id's entry in the shape `..\u002F..\u002Freferences\u002Fmemory-schema.md`\ngives. Save the `capture` output (step 2) and the `ids` output (step 3) as files\nin `\u003Cruns-dir>`; the command reads the patch path, hash and files from the first\nand, for a deselecting candidate, `removed_ids` and `removes_tests` from the\nsecond. Add the measured fields with `--set`:\n\n```console\n$ python3 \u003Cmeasure.py> record \u003Cmemory-dir>\u002Fbenchmarks.json \\\n    --state \u003Cmemory-dir>\u002Fstate.json \\\n    --id \u003Chypothesis-id> \\\n    --capture-json \u003Cruns-dir>\u002F\u003Chypothesis-id>-capture.json \\\n    --ids-json \u003Cruns-dir>\u002F\u003Chypothesis-id>-ids.json \\\n    --set 'median_delta=\u003Cseconds>' \\\n    --set 'clears_noise=\u003Ctrue|false>' \\\n    --set 'verdict=\"\u003Cvalidated|rejected>\"'\n```\n\nThe command stamps `env_token` and `base_head`, keeps the entries for ids this\nrun skipped (`--only` or a matching token) and replaces those for ids it\nbenchmarked, `--force` included. Set `confidence` high only for serial, repeated runs with the delta above the\nfloor. Nothing in this phase touches the user's tree, history, or any remote. Then mark the\nphase done, which clears the later flags:\n\n```console\n$ python3 \u003Cmeasure.py> state \\\n    complete \u003Cmemory-dir>\u002Fstate.json \\\n    --phase benchmark\n```\n\n## Step 4: Report\n\nEmit the `01-benchmark` sections from `..\u002F..\u002Freferences\u002Foutput-contract.md`. Write\nthe prose per `..\u002F..\u002Freferences\u002Fprose-rules.md`. Put each candidate under the\nsection its measurement earned: a candidate rejected for a count drop or a gate\nfailure is under `## Rejected`, never `## Validated`. An empty section says\n\"None.\" in one line. Advise nothing the runs did not measure (no \"make X\ncheaper\"). When the phase halted before any run, print the halted-run shape\ninstead, with one cause and no list of what is absent. Reply in this shape, as\nplain text outside any code fence, with plain bullets, no bold (never\n`- **Label:** ...`), and no local paths:\n\n```text\n✓ 1 validated, 1 rejected\n\n## Validated\n\n- H10 mis-scoped fixture: db_session to session scope saves 4.6s, above the 0.44s band, confidence 0.9.\n\n## Rejected\n\n- H33 fixed sleep in test_report: removing it saves 0.02s, inside the noise band.\n\n## Safety-gate results\n\n- H10 mis-scoped fixture: order independence held across 3 seeds and collection is deterministic.\n```\n\nClose with an `AskUserQuestion` panel offering to build the plan, re-benchmark a\nsubset, or stop. Without one, follow the contract's next-step rule.\n",{"name":5,"description":7,"allowed-tools":24,"argument-hint":32,"disable-model-invocation":33},[25,26,27,28,29,30,31],"Bash","Read","Grep","Glob","Write","Edit","AskUserQuestion","[--only=\u003Cheuristic-id>] [--runs=N] [--force] [--memory-dir=\u003Cpath>]",true,"main","5fbeb3b47beedea13813be7b7023f736a8d97a65","plugins\u002Fpytest-optimizer\u002Fskills\u002F01-benchmark","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills","https:\u002F\u002Fgithub.com\u002Ftony","https:\u002F\u002Ftony.sh","lighthouse keeper","2026-11-08 17:57:34",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":33},[44,47,50],{"tool":45,"command":46},"npx","npx skills add tony\u002Fskills --skill 01-benchmark",{"tool":48,"command":49},"bunx","bunx skills add tony\u002Fskills --skill 01-benchmark",{"tool":51,"command":52},"pnpm","pnpm dlx skills add tony\u002Fskills --skill 01-benchmark","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fpytest-optimizer\u002Fskills\u002F01-benchmark\u002FSKILL.md",[55],{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"agents\u002Fopenai.yaml","REFERENCE","参考文档",309,"https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fpytest-optimizer\u002Fskills\u002F01-benchmark\u002Fagents\u002Fopenai.yaml",[62,69,76,83,90,97],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7E8NCSHXV2JPW4J61R0TQS7","serene-bliss","Use when asked to make a CLI --help, docs page, error message, or tooling UX more serene or calmer for developer experience; returns text, no edits.","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fweave\u002Fskills\u002Fserene-bliss\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7E8N87MAVJD8RRQKQB1ZPS6","review","Weave code review: run independent adversarial reviews in parallel, then synthesize findings","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fweave\u002Fskills\u002Freview\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7E8N402XG4ZBFNACQ413163","refine","Use when asked to refine a draft, iterate on code until cleaner, or polish a README through repeated critique and improvement.","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fweave\u002Fskills\u002Frefine\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7E8N01KB86A7136P00GVKJW","prompt","Weave prompt: compare independent adversarial implementations, then adopt the single best one","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fweave\u002Fskills\u002Fprompt\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M7E8MW7YJW2PCMDW3G9RTGNK","plan","Weave planning: compare independent adversarial plans, then synthesize the best one","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fweave\u002Fskills\u002Fplan\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M7E8MSBYXNCV4ZYWDS4PFJXK","fix-review","Fix weave review findings: validate, add test coverage, fix, and commit each as atomic changes","https:\u002F\u002Fgithub.com\u002Ftony\u002Fskills\u002Fblob\u002Fmain\u002Fplugins\u002Fweave\u002Fskills\u002Ffix-review\u002FSKILL.md",[],[]]