mutation-audit
Measuring whether UZE's tests would notice the code being wrong, with cargo-mutants - starting and monitoring `mutants.yml`, downloading the evidence, triaging each surviving mutant, writing the test a real gap needs and proving it catches the mutant. Use when someone asks about mutation testing, mutants, survivors, `missed.txt` or a "Missed mutant" annotation, whether tests really prove anything or only pass ("os testes testam mesmo?", "teste que só passa"), to audit a crate's tests, to check tests an agent wrote, or before refactoring code whose tests nobody has measured. Not for reviewing a whole change (that is `code:polish`), not for a coverage figure alone, and not for a security audit (that is `security-audit`).
- License 许可
- Apache-2.0
- 所属仓库
- uze-sh/uze
- 最近更新
- 2026-11-13
- Stars
- ★ 4
- Forks
- 1
01 前言 Front Matter
- name
- mutation-audit
- description
- Measuring whether UZE's tests would notice the code being wrong, with cargo-mutants - starting and monitoring `mutants.yml`, downloading the evidence, triaging each surviving mutant, writing the test a real gap needs and proving it catches the mutant. Use when someone asks about mutation testing, mutants, survivors, `missed.txt` or a "Missed mutant" annotation, whether tests really prove anything or only pass ("os testes testam mesmo?", "teste que só passa"), to audit a crate's tests, to check tests an agent wrote, or before refactoring code whose tests nobody has measured. Not for reviewing a whole change (that is `code:polish`), not for a coverage figure alone, and not for a security audit (that is `security-audit`).
- compatibility
- The GitHub CLI for runs; for the local lab, Linux with bubblewrap (bwrap), cargo and this repository's `security-audit` skill, whose sandbox it uses.
02 资源文件 1
- 脚本scripts/lab4.0 KB
03 SKILL.md 正文
Mutation audit
Coverage says a line ran. A mutant asks whether a test would notice that line being wrong. The output of an audit is a triaged list and the tests it led to, never a score.
The rules everything else serves
A test states behavior, and the mutant only points at it. Read what the
code should do from its doc comment, docs/architecture/invariants.md, the
OpenSpec specs or the rule that owns it. A test copied from what the code does
today locks in whatever bug it has. If nothing says what is right, stop and ask;
that absence is a finding.
A survivor is a question. It may be a real gap, a mutant no test could catch, or something only another crate's tests would catch. Read the code before concluding anything.
No target score. Some mutants are equivalent, so 100% is not the goal. Compare a crate with its own earlier runs, and never write a test only to move the number.
A claim is proven by re-running the mutant. "This test covers it" counts
once that mutant reads caught.
Before starting: results are public
The repository is public, so a run's logs, annotations, summary and artifacts are too. A survivor in a check names a check no test protects.
- Before dispatching a crate that holds trust, consent, confinement, approval or acquisition, confirm the operator accepts that, or use the lab below, which publishes nothing.
- If triage finds an exploitable flaw and not only a missing test, stop and
follow the
security-findingsrule. It enters the repository with its fix, and the commit, test name and PR text do not explain the attack before then.
The loop
1. Measure
gh workflow run mutants.yml -f package=<crate> # that crate's tests
gh workflow run mutants.yml -f package=<crate> -f test_scope=workspace
gh workflow run mutants.yml -f package=<crate> -f mode=diff -f base=<tag> -f ref=<branch>
Scope crate is a cheap list of candidates and overstates survivors, because a
foundation crate's behavior is often tested by whoever uses it. Scope workspace
is the verdict on whether the project as a whole notices. Use workspace for a
small crate and crate first for a large one, then confirm what matters in the
lab. In scope crate, the survivors of uze-contract/src/docs.rs mean nothing: it
compiles only with feature schema, which that scope leaves off. mode=diff suits a branch or a release range. ref defaults to main, so
name the branch you are auditing, or the run measures something else and succeeds.
Start one crate at a time: GitHub keeps one run waiting, and a third start
replaces it. A run shares the account's 20 concurrent jobs with ci.yml, so
cancel it (gh run cancel <id>) when a delivery needs them.
Measured so far, for planning a run rather than as a promise:
| Run | Mutants | Median per mutant | Runner-hours | Wall clock |
|---|---|---|---|---|
uze-document, scope crate |
69 | 1 s | 0.3 | 20 min |
uze-document, scope workspace |
69 | 97 s | 2.6 | 48 min |
A mutant in a foundation crate rebuilds what depends on it, so scope workspace
costs most there.
2. Collect
gh run watch <id> --exit-status
gh run download <id> --pattern 'mutants-*' --dir <scratch>
python3 .github/scripts/mutants.py report <scratch> \
--package <crate> --planned <n> --commit <sha> --missed <scratch>/missed.txt
The run's summary has the same report. Annotations are only a sample of the
survivors, so read missed.txt. Incomplete means shards ran out of time:
raise -f shards=<n> before reading the score. A job's success says the run
worked, and survivors do not fail it.
3. Triage
Open each survivor at its line and put it in one row. Have someone who did not
write the code judge it (the code:reviewer agent), since whoever wrote it reads
what they meant.
| Verdict | How to recognize it | What to do |
|---|---|---|
| Real gap | The change would alter something a user or a later check relies on | Step 4 |
| Caught elsewhere | Scope was crate, and the behavior is exercised by a consumer |
Re-run in scope workspace |
| Equivalent | The mutant cannot change behavior: an unreachable branch, a value equal to what it replaces | Record it in .cargo/mutants.toml with the reason |
| Not worth testing | A log line, a debug message, cosmetic output | Exclude it with the reason, if it recurs |
Patterns from past audits:
-> bool with trueandwith falseboth surviving: nothing checks the answer. It is the strongest signal on the list.- Only
with truesurviving: every test sees the true case, and nobody asked whether the function ever says no. replace > with >=: no test sits on the boundary. First check whether equality can reach that line at all, since an earlier return makes the mutant equivalent.replace f -> Vec<_> with vec![]on something a check iterates: the check still passes, having checked nothing.- A match guard replaced
with true: a narrow error handler became a catch-all. Failure paths are where this hides, as a test rarely forces a read to fail.
4. Write the test
Name the behavior, and say in a comment why it matters only when that is not obvious. The test belongs with the code it proves, in the crate that owns the rule, even if a consumer already covers part of it. Don't assert an implementation detail or duplicate the expression under test:
assert!(may_vote(18)); // states the rule
assert_eq!(may_vote(18), 18 >= 18); // copies the code and proves nothing
A test for a bug the mutant exposed lands with the fix, in a fix: commit. A test
for correct code is a test: commit.
5. Prove it
Run the lab over the changed files, which leaves a list that should be shorter by
exactly what you meant to catch. scripts/lab is relative to this skill, and the
script works from any directory:
AUDIT=<dir> scripts/lab --package <crate> --file '<path>'
<dir> sits on a disk, outside the repository: the builds outgrow a memory-backed
/tmp, which everything else on the machine shares, so the lab refuses a tmpfs.
Add --test-workspace=true to judge against the whole workspace, which first proves
the suite green in the box, as cargo-mutants' own baseline would try only the crate, and --re '<pattern>' to re-run one mutant. Read
$AUDIT/boxes/mutants/work/uze/mutants.out/missed.txt; the file is what proves it.
The box can mislead or get in the way in four ways:
- A false
missed. Bun is absent, and the OpenCode tests that need it skip themselves. Before calling a survivor in that code a gap, confirm it on a CI run, which installs Bun. - A false
caught. A test that fails now and then there reads as catching the mutant. When a result surprises you, read that mutant's log undermutants.out/log/to see which test failed. The timing budgets (performance,extension_storm,budget) fail on a busy machine, so a workspace run leaves them out; a mutant only they would catch reads asmissed. - The machine runs out. The box limits neither memory nor CPU. A mutant of
uze-scriptcan take a limit off the engine, and one test reached 33 GB. Watch the available memory while that crate runs. - No result. A test that always fails there stops the run at the baseline.
Find out why it fails: anything in a sandbox is refused an approval, for example,
so those tests are filtered out for a workspace run in
scripts/lab. A bare--baseline=skipwould hide it by making every mutant readcaught.
A caught is only as good as the test that failed. If one test catches unrelated
mutants, or a report is too good to be true, read the logs first.
6. Deliver
Open an ordinary pull request. Its body says which mutants the tests now catch and how many remain, with the triage for what remains, and names no attack. Comparing before and after is the evidence. Say what was not run.
Recording an equivalent mutant
.cargo/mutants.toml is the list of what was read and set aside. Match on the file
and the change, without the line number, which moves:
# `read` has already returned when the shape equals SHAPE.
'uze-document/src/lib\.rs.*replace > with >= in read$',
Check it with AUDIT=<dir> scripts/lab --list --package <crate> | wc -l.
An exclusion
without its reason gets deleted, and anyone changing the code it names re-reads
it. It is for equivalents and noise, never for a real gap someone would rather not
fix.
Never
- Run
cargo mutantsoutside the lab on this machine. A mutant can take a test out of its isolation, anduze-platformanduze-terminalhold code that ends processes (theending-processesrule). - Add a mutation run to the
Gate. It is slow and it reports questions. - Set a mutant timeout close to the suite's own length. A mutant that outlasts the limit counts as detected, so an honest slow run would inflate the score.
- Delete or weaken an assertion to make a report tidier.
04 同仓库更多 Skills
commit
North Star fixture skill — used to prove native/generated delivery across harnesses.
oc-skill
opencode native
uze-plugin-first
Emits the deterministic UZE Plugin First conformance proof.
hello
says hello
agy-native
antigravity native
review
Review code for correctness and missing tests