noyalib-validation-and-qa
What counts as evidence in noyalib. Use before opening a PR, claiming a fix is done, or adding a test: the evidence bar, test taxonomy, doctest rules, cross-loader parity regression tests, acceptance thresholds. Not running tools (noyalib-diagnostics-and-tooling).
- License 许可
- MIT
- 最近更新
- 2026-11-09
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- noyalib-validation-and-qa
- license
- MIT
- metadata
- { "agtmls-owner": "Sebastien Rousseau", "agtmls-bundle": "noyalib", "agtmls-maturity": "project", "agtmls-risk-level": "medium", "agtmls-writes-files": "true", "agtmls-network-access": "optional", "agtmls-handles-secrets": "false", "agtmls-executes-commands": "true", "agtmls-requires-human-review": "true" }
- description
- What counts as evidence in noyalib. Use before opening a PR, claiming a fix is done, or adding a test: the evidence bar, test taxonomy, doctest rules, cross-loader parity regression tests, acceptance thresholds. Not running tools (noyalib-diagnostics-and-tooling).
- allowed-tools
- Read Glob Grep Write Edit Bash WebFetch WebSearch
- compatibility
- Requires cargo, git. Tested with Claude Code, Codex, and Aider skill layouts
02 资源文件 2
- 参考文档metadata.json411 B
- 参考文档reference.md7.0 KB
03 SKILL.md 正文
noyalib Validation & QA
The evidence bar. A PR that "looks right" is not merged; a PR that proves a behaviour with a test that would have failed on the previous tree is. This skill fences that discipline into checklists.
- Audience: mid-level engineer or Sonnet-class model.
- Scope: pure-Rust YAML library
noyalibv0.0.14, test harness and conformance suites. - Not this skill:
- Running / interpreting the tools →
noyalib-diagnostics-and-tooling. - The 95 → 98 coverage push specifically →
noyalib-coverage-campaign. - Chronicle of settled fights →
noyalib-failure-archaeology.
- Running / interpreting the tools →
- Date-stamped: 2026-07-07.
Reference material (full 130-file test taxonomy, the official YAML Test Suite mechanics + bench-fixture inventory, and the read-only provenance one-liners): see
reference.mdin this directory. This file keeps the evidence bar, the add-a-test checklists, the acceptance thresholds, and what does NOT count.
1. The evidence bar (non-negotiable)
A fix is proven by a test that FAILS on the pre-fix tree. The
discipline is: write the regression test first, watch it fail
(cargo test <name> red), then fix, then watch green. CONTRIBUTING.md
lines 47–48 codify the rule: "Add or update tests in the same commit /
PR as the behaviour change. Never as a follow-up."
There is one caveat driven by the [CI-always-green invariant] captured
in MEMORY.md: red tests never get PUSHED alone. The red→green
transition happens locally. What lands on main is the fix plus its
regression test, both green, in one commit. A reviewer reconstructs the
red state mentally by reading the test and the fix together — the test
name typically encodes the pre-fix bug (e.g.
from_str_value_refuses_distinct_typed_key_collision).
"Wrong-but-green" is the enemy. The silent key-collision collapse
survived 4000+ green tests because no test compared the two loader
paths against the same input. See
tests/no_span_loader_parity.rs:1–43 — it now pins that any semantics
shared by two code paths must have a cross-path test. The rule
generalises:
If a behaviour is implemented by more than one code path (span-full loader vs
NoSpanLoader, streaming vs blocking, borrowed vs owned, SIMD vs scalar), the invariant needs a test that exercises both paths against the same input and asserts they agree.
Exemplars to copy:
tests/no_span_loader_parity.rs— loader parity.tests/simd_equivalence.rs— SIMD/scalar path parity.tests/error_kind.rs— pins theError → ErrorKindclassifier (downstream consumers routing on kind).
2. Test taxonomy
crates/noyalib/tests/ holds 130 top-level integration test files
(ls crates/noyalib/tests/*.rs | wc -l), every one SPDX-headed, plus
~480 doctests. When adding a file, match an existing family:
coverage_* (llvm-cov gaps), cst_* (green-tree/round-trip/span),
spec/<area>.rs, de*/ser*/serde* (serde surface), issue_<n>
(incident regressions), competitor_bugs.rs, proptest.rs, and the
*_panic_regressions.rs fuzz corpora. Full family-by-family table with
file counts in reference.md §R1.
3. Certified / golden inventory
The conformance gate is tests/official_suite.rs — asserts
compliance >= 94.0 with an empty SKIP_LIST; the README
"387/387 / 19 skipped" and BENCHMARKS.md "406/406" numbers are
documentation drift, so cite the test files (351 wrapper .yaml cases,
94% floor), not the prose. Any loaded-case pass-count drop is a release
blocker. Golden bench fixtures live in benches/fixtures/*.yaml — do
not mutate without a paired Criterion baseline. Full mechanics (marker
decoder, verbatim assertion arithmetic, companion report, fixture
table) in reference.md §R2.
4. How to add a test — checklists per kind
4.1 Integration test (new .rs file under tests/)
-
Pick the family (
coverage_*,cst_*,de_*,issue_<n>, spec/, competitive_*, etc.) and match its naming. New file needs the exact SPDX header every existing test carries:// SPDX-License-Identifier: MIT OR Apache-2.0 // Copyright (c) 2026 Noyalib. All rights reserved. //! One-line summary — what invariant this file locks. //! //! Longer context: what was wrong, or what path this covers.(Verify shape against any existing file, e.g.
tests/issue_46.rsortests/no_span_loader_parity.rs.) -
If it's a bug regression, name the test after the bug and start the file's module doc with
//! Regression test for …. -
If it's spec-area coverage, put it under
tests/spec/<area>.rsand register the module intests/spec/mod.rs. -
Feature-gated tests need either
required-featuresinCargo.toml's[[test]]entry, or#[cfg(feature = "…")]inside the file. Prefer thecfggate — no manifest churn. -
Anything touching two code paths (blocking vs streaming, spanned vs no-span, SIMD vs scalar) needs a cross-path parity test — see
tests/no_span_loader_parity.rsandtests/simd_equivalence.rs.
4.2 Spec case
- One
#[test] fn <snake_case_name>()per assertion. - File already exists (
tests/spec/scalars.rsetc.) → add the test there. If the area has no file yet, create it undertests/spec/and wire it intotests/spec/mod.rs. - Use
noyalib::from_str(or the specific loader under test) directly; keep the assertion small — one input, one expected value.
4.3 Doctest
-
Every public item ships a
# Examplesblock.CONTRIBUTING.md:116makes this explicit. -
For fallible APIs also add
# Errors. -
Errors-parse-a-repro pattern: a doctest that demonstrates an error variant must parse an actual reproducer YAML, not just construct the variant. Exemplar (
src/error.rs:504–508):/// ``` /// use noyalib::{Error, Value, from_str}; /// let err = from_str::<Value>("1: a\n\"1\": b\n").unwrap_err(); /// assert!(matches!(err, Error::KeyCollision(_))); /// ```Constructing
Error::KeyCollision("1".into())on its own is allowed only as a supplementary shape-example; the behaviour demonstration must parse.
4.4 Property test
- Add to
tests/proptest.rs(or a newproperties_<area>.rs). - The universal property must be readable in one sentence
(
parse(emit(v)) == v,parse(format(s))is total, etc.). - Shrink to minimal counterexamples; commit those as a fixed regression
test in the nearest topical
tests/*.rsfile.
4.5 Fuzz-discovered regression
- Add the minimised crashing input to
tests/scanner_panic_regressions.rs(or the appropriate*_panic_regressions.rs) — one#[test]per input. - Keep the raw corpus entry under
fuzz/corpus/<target>/so the fuzzer will not re-discover it.
5. Acceptance thresholds
Before you claim "ready to merge":
| Gate | Threshold | Where enforced |
|---|---|---|
| Official YAML Test Suite | compliance >= 94.0 assertion; SKIP_LIST = &[] so total-loaded pass count must not drop |
tests/official_suite.rs:295–314 + review |
| Coverage — functions | Target 98% / current gate 96% | --fail-under-functions 96 (ci.yml:160 + shared-coverage.yml default) |
| Coverage — lines | Target 98% / current gate 94% | --fail-under-lines 94 (ci.yml:161 + shared-coverage.yml default) |
| Coverage — regions | Target 98% / current gate 93% | --fail-under-regions 93 (ci.yml:162 + shared-coverage.yml default) |
| Clippy | -D warnings — zero warnings |
Workspace CI |
| Rustfmt | Clean (make fmt) |
Workspace CI |
| Doctests | All ~480 pass (cargo test --doc) |
Workspace CI |
| REUSE / SPDX | Every source file carries a header | REUSE workflow |
| Miri | Focused pass green per PR | Miri workflow |
| Fuzz smoke | Differential fuzz 10s per target | Differential fuzz workflow |
The 98/98/98 target is the live campaign — see
noyalib-coverage-campaign for the plan to lift the live gate
(96/94/93 functions/lines/regions) up to 98/98/98. doc/TESTING.md
still cites 95 / 93 / 92; that is documentation drift — trust
the workflow files. A conformance regression (any yaml-test-suite
case dropping from pass to fail/skip) is a release blocker,
not a "next PR" item.
6. What does NOT count as evidence
Explicitly rejected. If any of the below is your entire justification, the PR is not ready:
- "It looks right." — reading the diff is not testing.
- A passing test with no failing-before counterpart. — the test must be shown (locally or via reviewer's mental replay of the diff) to fail on the pre-fix tree.
- Coverage-only tests that execute but do not assert. — every
#[test]asserts something an engineer can read. A test that only callsfrom_strand drops the result is not a test; wrap it withassert!(matches!(...))orassert_eq!on a value. Thecoverage_*family is not exempt from this rule. - Bench deltas within noise. — Criterion prints a p-value; if the CI comparison harness (CodSpeed) does not flag the delta as significant, it is not a regression and it is not an improvement. Do not claim either.
- "Green on my machine." — full CI must be green. See the CI-always-green invariant.
- A doctest that only constructs a type. — for behavioural claims, parse a reproducer (see §4.3).
7. When NOT to use this skill
- You need to run the tests or interpret llvm-cov output →
noyalib-diagnostics-and-tooling. - You are working the 95 → 98 coverage push specifically →
noyalib-coverage-campaign. - You are researching whether a bug was fixed before →
noyalib-failure-archaeology. - You are deciding what to build →
noyalib-architecture-contract.
Provenance
The read-only re-verification one-liners (test-file count, SPDX
coverage, suite case count, the coded compliance >= floor, doctest
count, wired coverage thresholds, fixture inventory) live in
reference.md §R3. Run them before quoting any number here.
Date-stamp: 2026-07-07 · Crate version: v0.0.14 · Ground
truth: crates/noyalib/tests/, doc/TESTING.md, CONTRIBUTING.md,
Makefile.
04 同仓库更多 Skills
example-skill
Use when the user needs this reusable workflow. Replace this with trigger-rich wording that says when to load the skill, what task it handles, and when not to use it.
writing-plans
Turn a vague or multi-step request into an ordered list of small steps, each with its own done-condition and verification. Load before starting work that touches several files, has unclear scope, or spans more than one session, and when a request needs decomposing before any code is written. Produces a plan a reviewer can check against, not a to-do list.
web-research-and-source-triage
Use when gathering current web evidence, comparing sources, triaging credibility, reading GitHub repositories, checking docs, or deciding whether a claim needs live verification.
vetting-a-skill-before-install
Use before installing, importing or enabling a third-party skill, plugin, marketplace or skills repository in Claude Code, Codex, Antigravity, Cursor or any agent. Pins the source to a commit, audits every skill and the agent configuration it ships, reads what the audit could not cover, and ends in install, mirror or refuse. Not for auditing this registry's own skills.
verification-before-completion
The gate between doing work and claiming it is done. Load before saying finished, fixed, working, or ready, before marking a task complete, and before reporting an outcome to a human. Converts each claim into the observation that would justify it, and names the rationalizations that let unverified work ship. Not the project's own evidence bar or test taxonomy.
using-agtmls
Meta-router for the AgtMLS skill catalog. Read first when unsure which AgtMLS skill applies, when several look relevant, when asking 'is there a skill for porting or noyalib work', or when onboarding to how the hub's skills and shared references fit together.