[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7E4Q4BX1EWPD0C71V3CDARR":3,"skill-same:01M7E4Q4BX1EWPD0C71V3CDARR":63},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7E4Q4BX1EWPD0C71V3CDARR","flaky-test-hunt",null,"Find why a test or CI job fails only sometimes instead of retrying it: measure the failure rate, find and fix the cause, prove the fix with enough clean runs. Any language. Use when the user says 'this test is flaky', 'fails intermittently', 'passes locally but fails in CI', 'can we just retry it', 'quarantine this test', or wants to retry a failed CI run.","paureis\u002Fpau-skills","paureis","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F133926455?v=4","01M7E4HCQNKQ22F6VQFHRFHW8K","Alvaro Reis","01M7E4HCRJJ4EDXQHFJN1MW493",2,"MIT","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Fskills\u002Fverification\u002Fflaky-test-hunt\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: flaky-test-hunt\ndescription: \"Find why a test or CI job fails only sometimes instead of retrying it: measure the failure rate, find and fix the cause, prove the fix with enough clean runs. Any language. Use when the user says 'this test is flaky', 'fails intermittently', 'passes locally but fails in CI', 'can we just retry it', 'quarantine this test', or wants to retry a failed CI run.\"\nargument-hint: \"\u003Ctest id, failing CI job or run link, and anything known about how often it fails>\"\n---\n\n# Flaky test hunt\n\n\"Flaky\" describes a symptom, not a cause. A test that fails some of the time is reporting a real nondeterminism:\nin the test, in its environment, or in the product. This skill finds which one, fixes it, and proves the fix with\nnumbers rather than with one green run.\n\nContext from the user: $ARGUMENTS\n\nReference files, read when you reach their phase:\n- `${CLAUDE_SKILL_DIR}\u002FCOMMANDS.md`: per-ecosystem commands to isolate, repeat, shuffle and parallelize\n- `${CLAUDE_SKILL_DIR}\u002FTAXONOMY.md`: the causes, each with symptoms, how to confirm, right and wrong fix\n- `${CLAUDE_SKILL_DIR}\u002Frepeat.mjs`: runs any command N times and reports the failure rate,\n  a 95% interval and the distinct failure signatures\n\n## Rules that apply throughout\n\n- **Do not \"fix\" by retrying, waiting longer, or hiding the test.** Adding retries (`jest.retryTimes`, pytest\n  `--reruns`, Surefire `rerunFailingTestsCount`, a CI retry step), raising a timeout, adding or lengthening a sleep,\n  skipping, or quarantining are not fixes. Each one makes the signal quieter while the cause stays, and the cause is\n  sometimes a real product bug that users will hit. You may propose one of these as a temporary measure, clearly\n  labelled as such, only with the user's explicit approval, and only alongside a ticket for the real fix.\n- **Do not call anything \"flaky\" in the report without a category from `TAXONOMY.md` and the evidence for it.** If\n  you could not find the cause, say \"not reproduced\" or \"cause unknown\" and list what you tried and the counts.\n- **Do not change the assertion to make it pass** (loosening equality, widening a tolerance, deleting the check)\n  unless the analysis shows the assertion itself was wrong, and say so explicitly.\n- **Numbers, not impressions.** \"Ran it a few times and it passed\" is not evidence. Every claim about frequency\n  comes from a counted run: failures \u002F runs, with the mode it ran in.\n- **Change one thing at a time** between measured batches, so a change in the rate can be attributed.\n\n## Phase 1. Capture the evidence\n\nBefore running anything, collect what already exists. Read it from the repository and CI; ask the user only what\nthose cannot answer. For a CI failure, do \"Is it a flake at all?\" below first.\n\n- [ ] The exact test id (file, class, name, parameters) and the full failure output of at least one failing run:\n      assertion message, stack trace, and the lines before it. Save it; later phases compare against it.\n- [ ] How often: count failures of this test or job over recent CI history (for GitHub Actions, `gh run list\n      --workflow \u003Cfile> --limit 100 --json conclusion,headSha,createdAt` and `gh run view \u003Cid> --log-failed`). A rough\n      historical rate tells you how many local runs you will need to see it.\n- [ ] Where: which runners, OS images, architectures, language versions, shard or worker numbers, and time of day.\n      A failure that only appears on one shard, after midnight UTC, or on the macOS runner is already a strong clue.\n- [ ] What changed: when it started failing, and the commits, dependency bumps, CI image changes and test additions\n      around that date. `git log --since` on the test file, the code it covers and the test config.\n- [ ] Whether the failure output is the same every time. Two different messages are often two different problems.\n- [ ] The test runner and how CI invokes it (the actual command in the workflow, with its flags, env and parallelism).\n\n### Is it a flake at all?\n\nWhen the failure is in CI, and above all when the question is \"can we just retry it\" or \"it passes locally\", classify\nit before treating it as flaky, in this order:\n\n1. **Is the failing code in the diff?** Run `git fetch` first, so the commands below see the current base. If\n   `git rev-parse --is-shallow-repository` prints `true`, deepen first (`git fetch --deepen=\u003Cn>` or `--unshallow`):\n   a shallow clone gives no merge base, a wrong ancestry answer, and lists the boundary commit as touching every file.\n   Then compare the failing test and the code it covers with the branch's changes\n   (`git diff --name-only origin\u002F\u003Cbase>...HEAD`). If they do not overlap, suspect a stale base, even when the failure\n   reproduces locally: a branch that lacks the base's fix fails the same way everywhere.\n   - For a GitHub `pull_request` run (and any CI that tests a merged result), CI did not test the branch. It tested a\n     merge commit whose message names the base (`Merge \u003Chead> into \u003Cbase sha>`, in the checkout step's log), so check\n     for the base's fix against that base SHA, not against `origin\u002F\u003Cbase>`. The merge commit is not in an ordinary\n     clone: `git fetch origin pull\u002F\u003CN>\u002Fmerge` (or `git fetch origin \u003Csha>`); exit 128 means the server no longer has\n     it, so read the base SHA from the log instead. Also check whether the base branch itself is red at that SHA; if it\n     is, the failure is not this branch's.\n   - Otherwise run `git merge-base --is-ancestor origin\u002F\u003Cbase> HEAD`: exit 0 means the branch contains the base tip,\n     exit 1 means it is behind, so look at what the base has for the failing code\n     (`git log HEAD..origin\u002F\u003Cbase> -- \u003Cfiles>`).\n   A fix on the base means the answer is to update the branch, not to retry and not to patch outside the diff what the\n   base already fixed. Offer to bring the branch up to date the way the project does (merge or rebase), so CI runs\n   once on the result.\n2. **One re-run at most, as a diagnostic.** A single re-run shows whether the failure repeats. It is never the fix\n   (see the rules above). Running the same test locally is free and counts. A CI re-run costs runner minutes, so ask\n   before starting it unless the user asked for a re-run; a question such as \"can we just retry it?\" is not that\n   request. Do not re-run CI a second time.\n3. **Two identical failures are not proof of a consistent failure.** A test that fails one run in three fails twice\n   in a row one time in nine. Read the test's CI history first. End the hunt as a consistent failure only when history\n   shows no pass for this test at this commit and a few local runs (for example 3 of 3) fail the same way. Then say\n   what failed and stop, and offer to run `bug-fix` if it is installed; with or without it, ask before debugging.\n4. Everything else (a pass on a re-run of the same commit, a message that changes between runs, a history of\n   occasional failures) goes on to Phase 2.\n\nDone when you can state the failing test, one or more saved failure outputs, the historical frequency (or \"unknown\"),\nthe environments it fails in, and the CI command, and, for a CI failure, which of the cases above it is.\n\n## Phase 2. Reproduce by repetition, with a measured rate\n\nDetect the project's own runner first (lockfiles, `package.json` scripts, `pyproject.toml`, `go.mod`, `Cargo.toml`,\nbuild files, the CI workflow) and use the commands in `COMMANDS.md` for it. Then run the ladder below, recording\n`failures \u002F runs` for each rung. Use `repeat.mjs` to do the counting, so the numbers and the failure signatures are\nnot tallied by hand:\n\n```bash\nnode \"${CLAUDE_SKILL_DIR}\u002Frepeat.mjs\" --runs 50 --timeout 120 --json flake-isolated.json -- pytest \"tests\u002Ftest_x.py::test_y\" -p no:randomly\n```\n\nIt sets `REPEAT_RUN` in each run's environment, groups failures by a normalized signature of their last lines, and\nexits 1 if any run failed. Add `--parallel K` to run copies at once, `--until-fail` to stop at the first failure, and\n`--shell` for npm\u002Fnpx on Windows, pipes or `&&`.\n\n| Rung | What runs | What a failure here suggests |\n|---|---|---|\n| 1. Isolated | the single test, N times, one process at a time | the cause is inside the test or the code it calls: time, randomness, async waits, races, network |\n| 2. With its file | the test's file or class, N times | another test in the same file leaks state into it |\n| 3. Whole suite | the full suite in CI order, fewer times (it is slower) | cross-file pollution, shared resources, resource exhaustion |\n| 4. Random order | suite shuffled with a recorded seed, several seeds | order dependence; a failing seed reproduces it on demand |\n| 5. Parallel | the suite or file with the CI's worker count, or more | shared ports, files, databases, globals across workers |\n\nPick N from the historical rate: to have a good chance (about 95%) of seeing a failure that happens with probability\np, you need about 3\u002Fp runs. A 10% flake needs about 30 runs; a 1% flake needs about 300, which usually means running\nthe isolated test in a loop rather than the suite.\n\nIf it does not reproduce locally, make the local run look like CI before raising N: same command and flags, same\nworker count, `CI=true`, same timezone and locale (`TZ=UTC`, `LANG=C.UTF-8`), fewer CPUs (`taskset -c 0` on Linux,\n`docker run --cpus=1`), background load, a clean checkout and cache. If it still does not reproduce, run the counting\nin CI itself (a manual workflow that runs `repeat.mjs` on the same runner image).\n\nDone when you have a table of rungs with `failures \u002F runs` each, at least one rung where it fails, and the failing\nseed if order matters. If no rung fails after the CI-like attempts, stop and report \"not reproduced\" with the table.\n\n## Phase 3. Classify the cause\n\nOpen `TAXONOMY.md` and match the evidence against the categories: order dependence and shared state, time and date\nboundaries, unseeded randomness, concurrency and races, async waits and sleeps, network and external services,\nresource exhaustion (ports, files, memory, handles), pollution of globals and environment variables, floating point,\nunordered collections, environment differences, and real product bugs that only appear under certain timing.\n\nThe rung where it first failed narrows the list. The failure signatures from `repeat.mjs` tell you whether there is\none cause or several; treat each distinct signature as its own hunt.\n\nThen **confirm** the category with the experiment `TAXONOMY.md` gives for it. The strongest confirmation turns the\nintermittent failure into a deterministic one: force the bad order, freeze the clock at the boundary, fix the seed\nthat fails, inject a delay at the suspected race point, block the network. A test that now fails 20\u002F20 under the\nforced condition and 0\u002F20 without it has a confirmed cause. Keep that forcing setup; it often becomes the regression\ntest.\n\nDone when one category is named, with the confirming experiment and its counts. \"Probably a race\" without a forced\nreproduction is a hypothesis, not a classification; say so if that is where you stopped.\n\n## Phase 4. Fix the root cause and prove it\n\nApply the right fix for the category from `TAXONOMY.md`, where the cause is, even in code the branch never touched.\nIf the cause is in the product (a real race, a missing await in application code, a date bug), fix the product and\ntell the user plainly that the test was right. If the fix chooses a product behaviour (a tie-break order, a rounding\nrule, which of two results wins), describe the choice and get the user's yes before the edit.\n\nThen prove it, in this order:\n\n1. The forced reproduction from Phase 3 now passes, or is kept as a regression test that fails without the fix\n   (revert the fix once to watch it fail, then restore it).\n2. Re-run the rung that reproduced it, in the same mode, until the clean count is large enough to mean something.\n3. Run the full suite once to check the fix broke nothing else.\n\nHow many clean runs are enough: if the failure rate before the fix was p, then 0 failures in n runs makes a remaining\nrate of p or more unlikely at about 95% confidence when n is about 3\u002Fp. This is the rule of three: with zero\nevents in n trials, the upper 95% bound on the rate is roughly 3\u002Fn. Examples:\n\n| Measured rate before | Clean runs needed (about 95%) |\n|---|---|\n| 20% | 15 |\n| 10% | 30 |\n| 5% | 60 |\n| 2% | 150 |\n| 1% | 300 |\n\nIf the \"before\" rate came from only a few failures, its interval is wide; use its lower end (printed by `repeat.mjs`)\nto pick n, which asks for more runs and is the honest choice. Twenty clean runs after a 5% flake proves almost\nnothing: a 5% flake passes 20 times in a row about a third of the time.\n\nDone when the report can say \"before: X\u002FN in mode M; after: 0\u002FN' in the same mode\", with N' at least about 3\u002Fp, and\nthe full suite green.\n\n## Phase 5. Report\n\nProduce this, filled in:\n\n```\n## Flaky test hunt: \u003Ctest id>\n\nSymptom:        \u003Cfailure message, one line> (saved output: \u003Cpath or excerpt>)\nHistory:        \u003Cfailures \u002F runs in CI over \u003Cperiod>, or \"unknown\">; environments: \u003Cwhere it failed>\nReproduction:\n  | Rung | Command (short) | Failures \u002F runs | 95% interval |\n  | ...  | ...             | ...             | ...          |\n  Failing seed \u002F condition: \u003Cseed, order, TZ, worker count, or \"none\">\nCategory:       \u003CTAXONOMY.md category>\nEvidence:       \u003Cthe confirming experiment and its counts, e.g. \"forced order A then B: 20\u002F20 fail; B alone: 0\u002F20\">\nRoot cause:     \u003Cfile:line and one or two sentences on the mechanism>\nFix:            \u003Cwhat changed and why it removes the cause; product or test>\nProof:          before \u003CX\u002FN> -> after \u003C0\u002FN'> in \u003Cmode>; regression test \u003Cname> fails without the fix: yes\u002Fno\nFull suite:     \u003Cpass\u002Ffail, command>\nNot done:       \u003Cother signatures seen, related tests with the same pattern, anything left for the user>\n```\n\nIf the cause was not found, the report says \"Cause: not found\" and keeps the reproduction table, the categories ruled\nout with the experiment that ruled each out, and what to try next. It does not say \"flaky\" as a conclusion.\n\n## When to stop and ask\n\n- The only fix you can find is a retry, a longer timeout, a skip or a quarantine: present the evidence and ask.\n- The cause is in a third-party service or in CI infrastructure the user controls but you cannot change.\n- Reproducing needs more runs than are reasonable locally (a 0.1% flake needs thousands): propose running the\n  counter in CI and estimate its cost first.\n","# Flaky test hunt\n\n\"Flaky\" describes a symptom, not a cause. A test that fails some of the time is reporting a real nondeterminism:\nin the test, in its environment, or in the product. This skill finds which one, fixes it, and proves the fix with\nnumbers rather than with one green run.\n\nContext from the user: $ARGUMENTS\n\nReference files, read when you reach their phase:\n- `${CLAUDE_SKILL_DIR}\u002FCOMMANDS.md`: per-ecosystem commands to isolate, repeat, shuffle and parallelize\n- `${CLAUDE_SKILL_DIR}\u002FTAXONOMY.md`: the causes, each with symptoms, how to confirm, right and wrong fix\n- `${CLAUDE_SKILL_DIR}\u002Frepeat.mjs`: runs any command N times and reports the failure rate,\n  a 95% interval and the distinct failure signatures\n\n## Rules that apply throughout\n\n- **Do not \"fix\" by retrying, waiting longer, or hiding the test.** Adding retries (`jest.retryTimes`, pytest\n  `--reruns`, Surefire `rerunFailingTestsCount`, a CI retry step), raising a timeout, adding or lengthening a sleep,\n  skipping, or quarantining are not fixes. Each one makes the signal quieter while the cause stays, and the cause is\n  sometimes a real product bug that users will hit. You may propose one of these as a temporary measure, clearly\n  labelled as such, only with the user's explicit approval, and only alongside a ticket for the real fix.\n- **Do not call anything \"flaky\" in the report without a category from `TAXONOMY.md` and the evidence for it.** If\n  you could not find the cause, say \"not reproduced\" or \"cause unknown\" and list what you tried and the counts.\n- **Do not change the assertion to make it pass** (loosening equality, widening a tolerance, deleting the check)\n  unless the analysis shows the assertion itself was wrong, and say so explicitly.\n- **Numbers, not impressions.** \"Ran it a few times and it passed\" is not evidence. Every claim about frequency\n  comes from a counted run: failures \u002F runs, with the mode it ran in.\n- **Change one thing at a time** between measured batches, so a change in the rate can be attributed.\n\n## Phase 1. Capture the evidence\n\nBefore running anything, collect what already exists. Read it from the repository and CI; ask the user only what\nthose cannot answer. For a CI failure, do \"Is it a flake at all?\" below first.\n\n- [ ] The exact test id (file, class, name, parameters) and the full failure output of at least one failing run:\n      assertion message, stack trace, and the lines before it. Save it; later phases compare against it.\n- [ ] How often: count failures of this test or job over recent CI history (for GitHub Actions, `gh run list\n      --workflow \u003Cfile> --limit 100 --json conclusion,headSha,createdAt` and `gh run view \u003Cid> --log-failed`). A rough\n      historical rate tells you how many local runs you will need to see it.\n- [ ] Where: which runners, OS images, architectures, language versions, shard or worker numbers, and time of day.\n      A failure that only appears on one shard, after midnight UTC, or on the macOS runner is already a strong clue.\n- [ ] What changed: when it started failing, and the commits, dependency bumps, CI image changes and test additions\n      around that date. `git log --since` on the test file, the code it covers and the test config.\n- [ ] Whether the failure output is the same every time. Two different messages are often two different problems.\n- [ ] The test runner and how CI invokes it (the actual command in the workflow, with its flags, env and parallelism).\n\n### Is it a flake at all?\n\nWhen the failure is in CI, and above all when the question is \"can we just retry it\" or \"it passes locally\", classify\nit before treating it as flaky, in this order:\n\n1. **Is the failing code in the diff?** Run `git fetch` first, so the commands below see the current base. If\n   `git rev-parse --is-shallow-repository` prints `true`, deepen first (`git fetch --deepen=\u003Cn>` or `--unshallow`):\n   a shallow clone gives no merge base, a wrong ancestry answer, and lists the boundary commit as touching every file.\n   Then compare the failing test and the code it covers with the branch's changes\n   (`git diff --name-only origin\u002F\u003Cbase>...HEAD`). If they do not overlap, suspect a stale base, even when the failure\n   reproduces locally: a branch that lacks the base's fix fails the same way everywhere.\n   - For a GitHub `pull_request` run (and any CI that tests a merged result), CI did not test the branch. It tested a\n     merge commit whose message names the base (`Merge \u003Chead> into \u003Cbase sha>`, in the checkout step's log), so check\n     for the base's fix against that base SHA, not against `origin\u002F\u003Cbase>`. The merge commit is not in an ordinary\n     clone: `git fetch origin pull\u002F\u003CN>\u002Fmerge` (or `git fetch origin \u003Csha>`); exit 128 means the server no longer has\n     it, so read the base SHA from the log instead. Also check whether the base branch itself is red at that SHA; if it\n     is, the failure is not this branch's.\n   - Otherwise run `git merge-base --is-ancestor origin\u002F\u003Cbase> HEAD`: exit 0 means the branch contains the base tip,\n     exit 1 means it is behind, so look at what the base has for the failing code\n     (`git log HEAD..origin\u002F\u003Cbase> -- \u003Cfiles>`).\n   A fix on the base means the answer is to update the branch, not to retry and not to patch outside the diff what the\n   base already fixed. Offer to bring the branch up to date the way the project does (merge or rebase), so CI runs\n   once on the result.\n2. **One re-run at most, as a diagnostic.** A single re-run shows whether the failure repeats. It is never the fix\n   (see the rules above). Running the same test locally is free and counts. A CI re-run costs runner minutes, so ask\n   before starting it unless the user asked for a re-run; a question such as \"can we just retry it?\" is not that\n   request. Do not re-run CI a second time.\n3. **Two identical failures are not proof of a consistent failure.** A test that fails one run in three fails twice\n   in a row one time in nine. Read the test's CI history first. End the hunt as a consistent failure only when history\n   shows no pass for this test at this commit and a few local runs (for example 3 of 3) fail the same way. Then say\n   what failed and stop, and offer to run `bug-fix` if it is installed; with or without it, ask before debugging.\n4. Everything else (a pass on a re-run of the same commit, a message that changes between runs, a history of\n   occasional failures) goes on to Phase 2.\n\nDone when you can state the failing test, one or more saved failure outputs, the historical frequency (or \"unknown\"),\nthe environments it fails in, and the CI command, and, for a CI failure, which of the cases above it is.\n\n## Phase 2. Reproduce by repetition, with a measured rate\n\nDetect the project's own runner first (lockfiles, `package.json` scripts, `pyproject.toml`, `go.mod`, `Cargo.toml`,\nbuild files, the CI workflow) and use the commands in `COMMANDS.md` for it. Then run the ladder below, recording\n`failures \u002F runs` for each rung. Use `repeat.mjs` to do the counting, so the numbers and the failure signatures are\nnot tallied by hand:\n\n```bash\nnode \"${CLAUDE_SKILL_DIR}\u002Frepeat.mjs\" --runs 50 --timeout 120 --json flake-isolated.json -- pytest \"tests\u002Ftest_x.py::test_y\" -p no:randomly\n```\n\nIt sets `REPEAT_RUN` in each run's environment, groups failures by a normalized signature of their last lines, and\nexits 1 if any run failed. Add `--parallel K` to run copies at once, `--until-fail` to stop at the first failure, and\n`--shell` for npm\u002Fnpx on Windows, pipes or `&&`.\n\n| Rung | What runs | What a failure here suggests |\n|---|---|---|\n| 1. Isolated | the single test, N times, one process at a time | the cause is inside the test or the code it calls: time, randomness, async waits, races, network |\n| 2. With its file | the test's file or class, N times | another test in the same file leaks state into it |\n| 3. Whole suite | the full suite in CI order, fewer times (it is slower) | cross-file pollution, shared resources, resource exhaustion |\n| 4. Random order | suite shuffled with a recorded seed, several seeds | order dependence; a failing seed reproduces it on demand |\n| 5. Parallel | the suite or file with the CI's worker count, or more | shared ports, files, databases, globals across workers |\n\nPick N from the historical rate: to have a good chance (about 95%) of seeing a failure that happens with probability\np, you need about 3\u002Fp runs. A 10% flake needs about 30 runs; a 1% flake needs about 300, which usually means running\nthe isolated test in a loop rather than the suite.\n\nIf it does not reproduce locally, make the local run look like CI before raising N: same command and flags, same\nworker count, `CI=true`, same timezone and locale (`TZ=UTC`, `LANG=C.UTF-8`), fewer CPUs (`taskset -c 0` on Linux,\n`docker run --cpus=1`), background load, a clean checkout and cache. If it still does not reproduce, run the counting\nin CI itself (a manual workflow that runs `repeat.mjs` on the same runner image).\n\nDone when you have a table of rungs with `failures \u002F runs` each, at least one rung where it fails, and the failing\nseed if order matters. If no rung fails after the CI-like attempts, stop and report \"not reproduced\" with the table.\n\n## Phase 3. Classify the cause\n\nOpen `TAXONOMY.md` and match the evidence against the categories: order dependence and shared state, time and date\nboundaries, unseeded randomness, concurrency and races, async waits and sleeps, network and external services,\nresource exhaustion (ports, files, memory, handles), pollution of globals and environment variables, floating point,\nunordered collections, environment differences, and real product bugs that only appear under certain timing.\n\nThe rung where it first failed narrows the list. The failure signatures from `repeat.mjs` tell you whether there is\none cause or several; treat each distinct signature as its own hunt.\n\nThen **confirm** the category with the experiment `TAXONOMY.md` gives for it. The strongest confirmation turns the\nintermittent failure into a deterministic one: force the bad order, freeze the clock at the boundary, fix the seed\nthat fails, inject a delay at the suspected race point, block the network. A test that now fails 20\u002F20 under the\nforced condition and 0\u002F20 without it has a confirmed cause. Keep that forcing setup; it often becomes the regression\ntest.\n\nDone when one category is named, with the confirming experiment and its counts. \"Probably a race\" without a forced\nreproduction is a hypothesis, not a classification; say so if that is where you stopped.\n\n## Phase 4. Fix the root cause and prove it\n\nApply the right fix for the category from `TAXONOMY.md`, where the cause is, even in code the branch never touched.\nIf the cause is in the product (a real race, a missing await in application code, a date bug), fix the product and\ntell the user plainly that the test was right. If the fix chooses a product behaviour (a tie-break order, a rounding\nrule, which of two results wins), describe the choice and get the user's yes before the edit.\n\nThen prove it, in this order:\n\n1. The forced reproduction from Phase 3 now passes, or is kept as a regression test that fails without the fix\n   (revert the fix once to watch it fail, then restore it).\n2. Re-run the rung that reproduced it, in the same mode, until the clean count is large enough to mean something.\n3. Run the full suite once to check the fix broke nothing else.\n\nHow many clean runs are enough: if the failure rate before the fix was p, then 0 failures in n runs makes a remaining\nrate of p or more unlikely at about 95% confidence when n is about 3\u002Fp. This is the rule of three: with zero\nevents in n trials, the upper 95% bound on the rate is roughly 3\u002Fn. Examples:\n\n| Measured rate before | Clean runs needed (about 95%) |\n|---|---|\n| 20% | 15 |\n| 10% | 30 |\n| 5% | 60 |\n| 2% | 150 |\n| 1% | 300 |\n\nIf the \"before\" rate came from only a few failures, its interval is wide; use its lower end (printed by `repeat.mjs`)\nto pick n, which asks for more runs and is the honest choice. Twenty clean runs after a 5% flake proves almost\nnothing: a 5% flake passes 20 times in a row about a third of the time.\n\nDone when the report can say \"before: X\u002FN in mode M; after: 0\u002FN' in the same mode\", with N' at least about 3\u002Fp, and\nthe full suite green.\n\n## Phase 5. Report\n\nProduce this, filled in:\n\n```\n## Flaky test hunt: \u003Ctest id>\n\nSymptom:        \u003Cfailure message, one line> (saved output: \u003Cpath or excerpt>)\nHistory:        \u003Cfailures \u002F runs in CI over \u003Cperiod>, or \"unknown\">; environments: \u003Cwhere it failed>\nReproduction:\n  | Rung | Command (short) | Failures \u002F runs | 95% interval |\n  | ...  | ...             | ...             | ...          |\n  Failing seed \u002F condition: \u003Cseed, order, TZ, worker count, or \"none\">\nCategory:       \u003CTAXONOMY.md category>\nEvidence:       \u003Cthe confirming experiment and its counts, e.g. \"forced order A then B: 20\u002F20 fail; B alone: 0\u002F20\">\nRoot cause:     \u003Cfile:line and one or two sentences on the mechanism>\nFix:            \u003Cwhat changed and why it removes the cause; product or test>\nProof:          before \u003CX\u002FN> -> after \u003C0\u002FN'> in \u003Cmode>; regression test \u003Cname> fails without the fix: yes\u002Fno\nFull suite:     \u003Cpass\u002Ffail, command>\nNot done:       \u003Cother signatures seen, related tests with the same pattern, anything left for the user>\n```\n\nIf the cause was not found, the report says \"Cause: not found\" and keeps the reproduction table, the categories ruled\nout with the experiment that ruled each out, and what to try next. It does not say \"flaky\" as a conclusion.\n\n## When to stop and ask\n\n- The only fix you can find is a retry, a longer timeout, a skip or a quarantine: present the evidence and ask.\n- The cause is in a third-party service or in CI infrastructure the user controls but you cannot change.\n- Reproducing needs more runs than are reasonable locally (a 0.1% flake needs thousands): propose running the\n  counter in CI and estimate its cost first.\n",{"name":5,"description":7,"argument-hint":24},"\u003Ctest id, failing CI job or run link, and anything known about how often it fails>","main","2817107c191e822dfe71d6aff5d286cc8211c592","skills\u002Fverification\u002Fflaky-test-hunt",0,"https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills","https:\u002F\u002Fgithub.com\u002Fpaureis","AI engineer | building multi-tenant products end to end","2026-11-12 06:02:49",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add paureis\u002Fpau-skills --skill flaky-test-hunt",{"tool":39,"command":40},"bunx","bunx skills add paureis\u002Fpau-skills --skill flaky-test-hunt",{"tool":42,"command":43},"pnpm","pnpm dlx skills add paureis\u002Fpau-skills --skill flaky-test-hunt","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Fskills\u002Fverification\u002Fflaky-test-hunt\u002FSKILL.md",true,[47,53,59],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"COMMANDS.md","REFERENCE","参考文档",9171,"https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Fskills\u002Fverification\u002Fflaky-test-hunt\u002FCOMMANDS.md",{"path":54,"resourceType":55,"resourceTypeDesc":56,"sizeBytes":57,"sourceHtmlUrl":58},"repeat.mjs","SCRIPT","脚本",14186,"https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Fskills\u002Fverification\u002Fflaky-test-hunt\u002Frepeat.mjs",{"path":60,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":61,"sourceHtmlUrl":62},"TAXONOMY.md",15040,"https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Fskills\u002Fverification\u002Fflaky-test-hunt\u002FTAXONOMY.md",[64,71,76,83,89,96],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M7E4SRKNACXVPZYJ6HX2PQXY","tidy-notes","Tidy a plain text list: sort the lines and remove repeated ones. Use when the user types \u002Ftidy-notes.","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskill-eval\u002Fvariant-b\u002FSKILL.md",[],[],{"skillNo":72,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7E4SMEGAVM1547Y611T38SM","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskill-eval\u002Fvariant-a\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7E4SGSPHAJKYR8RX45ZKH3F","other-notes","A skill with a different name, for the refusal case.","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskill-eval\u002Fother-name\u002FSKILL.md",[],[],{"skillNo":84,"name":66,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M7E4SCTQ7AR5MGPJ3YRXSF06","Tidy a plain text list: sort the lines and remove repeated ones. Use when the user asks to tidy notes.","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskill-eval-triggers\u002Fskills\u002Ftidy-notes\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M7E4S920AVTNN4W61JQY4X9Z","pack-boxes","Plan how to pack boxes for a move, room by room. Use when the user asks to pack for a move.","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskill-eval-triggers\u002Fskills\u002Fpack-boxes\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M7E4S4ZXZWND0TGQ1E4N39KP","renamed-skill","A skill whose folder name differs from its name. Use when the user asks to rename something.","https:\u002F\u002Fgithub.com\u002Fpaureis\u002Fpau-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskill-eval-triggers\u002Frenamed\u002Fsome-folder\u002FSKILL.md",[],[]]