AGENT SKILL

noyalib-validation-and-qa

What counts as evidence in noyalib. Use before opening a PR, claiming a fix is done, or adding a test: the evidence bar, test taxonomy, doctest rules, cross-loader parity regression tests, acceptance thresholds. Not running tools (noyalib-diagnostics-and-tooling).

License 许可
MIT
最近更新
2026-11-09
Stars
★ 1
Forks
0

01 前言 Front Matter

name
noyalib-validation-and-qa
license
MIT
metadata
{ "agtmls-owner": "Sebastien Rousseau", "agtmls-bundle": "noyalib", "agtmls-maturity": "project", "agtmls-risk-level": "medium", "agtmls-writes-files": "true", "agtmls-network-access": "optional", "agtmls-handles-secrets": "false", "agtmls-executes-commands": "true", "agtmls-requires-human-review": "true" }
description
What counts as evidence in noyalib. Use before opening a PR, claiming a fix is done, or adding a test: the evidence bar, test taxonomy, doctest rules, cross-loader parity regression tests, acceptance thresholds. Not running tools (noyalib-diagnostics-and-tooling).
allowed-tools
Read Glob Grep Write Edit Bash WebFetch WebSearch
compatibility
Requires cargo, git. Tested with Claude Code, Codex, and Aider skill layouts

02 资源文件 2

03 SKILL.md 正文

noyalib Validation & QA

The evidence bar. A PR that "looks right" is not merged; a PR that proves a behaviour with a test that would have failed on the previous tree is. This skill fences that discipline into checklists.

  • Audience: mid-level engineer or Sonnet-class model.
  • Scope: pure-Rust YAML library noyalib v0.0.14, test harness and conformance suites.
  • Not this skill:
    • Running / interpreting the tools → noyalib-diagnostics-and-tooling.
    • The 95 → 98 coverage push specifically → noyalib-coverage-campaign.
    • Chronicle of settled fights → noyalib-failure-archaeology.
  • Date-stamped: 2026-07-07.

Reference material (full 130-file test taxonomy, the official YAML Test Suite mechanics + bench-fixture inventory, and the read-only provenance one-liners): see reference.md in this directory. This file keeps the evidence bar, the add-a-test checklists, the acceptance thresholds, and what does NOT count.


1. The evidence bar (non-negotiable)

A fix is proven by a test that FAILS on the pre-fix tree. The discipline is: write the regression test first, watch it fail (cargo test <name> red), then fix, then watch green. CONTRIBUTING.md lines 47–48 codify the rule: "Add or update tests in the same commit / PR as the behaviour change. Never as a follow-up."

There is one caveat driven by the [CI-always-green invariant] captured in MEMORY.md: red tests never get PUSHED alone. The red→green transition happens locally. What lands on main is the fix plus its regression test, both green, in one commit. A reviewer reconstructs the red state mentally by reading the test and the fix together — the test name typically encodes the pre-fix bug (e.g. from_str_value_refuses_distinct_typed_key_collision).

"Wrong-but-green" is the enemy. The silent key-collision collapse survived 4000+ green tests because no test compared the two loader paths against the same input. See tests/no_span_loader_parity.rs:1–43 — it now pins that any semantics shared by two code paths must have a cross-path test. The rule generalises:

If a behaviour is implemented by more than one code path (span-full loader vs NoSpanLoader, streaming vs blocking, borrowed vs owned, SIMD vs scalar), the invariant needs a test that exercises both paths against the same input and asserts they agree.

Exemplars to copy:

  • tests/no_span_loader_parity.rs — loader parity.
  • tests/simd_equivalence.rs — SIMD/scalar path parity.
  • tests/error_kind.rs — pins the Error → ErrorKind classifier (downstream consumers routing on kind).

2. Test taxonomy

crates/noyalib/tests/ holds 130 top-level integration test files (ls crates/noyalib/tests/*.rs | wc -l), every one SPDX-headed, plus ~480 doctests. When adding a file, match an existing family: coverage_* (llvm-cov gaps), cst_* (green-tree/round-trip/span), spec/<area>.rs, de*/ser*/serde* (serde surface), issue_<n> (incident regressions), competitor_bugs.rs, proptest.rs, and the *_panic_regressions.rs fuzz corpora. Full family-by-family table with file counts in reference.md §R1.


3. Certified / golden inventory

The conformance gate is tests/official_suite.rs — asserts compliance >= 94.0 with an empty SKIP_LIST; the README "387/387 / 19 skipped" and BENCHMARKS.md "406/406" numbers are documentation drift, so cite the test files (351 wrapper .yaml cases, 94% floor), not the prose. Any loaded-case pass-count drop is a release blocker. Golden bench fixtures live in benches/fixtures/*.yaml — do not mutate without a paired Criterion baseline. Full mechanics (marker decoder, verbatim assertion arithmetic, companion report, fixture table) in reference.md §R2.


4. How to add a test — checklists per kind

4.1 Integration test (new .rs file under tests/)

  1. Pick the family (coverage_*, cst_*, de_*, issue_<n>, spec/, competitive_*, etc.) and match its naming. New file needs the exact SPDX header every existing test carries:

    // SPDX-License-Identifier: MIT OR Apache-2.0
    // Copyright (c) 2026 Noyalib. All rights reserved.
    
    //! One-line summary — what invariant this file locks.
    //!
    //! Longer context: what was wrong, or what path this covers.
    

    (Verify shape against any existing file, e.g. tests/issue_46.rs or tests/no_span_loader_parity.rs.)

  2. If it's a bug regression, name the test after the bug and start the file's module doc with //! Regression test for ….

  3. If it's spec-area coverage, put it under tests/spec/<area>.rs and register the module in tests/spec/mod.rs.

  4. Feature-gated tests need either required-features in Cargo.toml's [[test]] entry, or #[cfg(feature = "…")] inside the file. Prefer the cfg gate — no manifest churn.

  5. Anything touching two code paths (blocking vs streaming, spanned vs no-span, SIMD vs scalar) needs a cross-path parity test — see tests/no_span_loader_parity.rs and tests/simd_equivalence.rs.

4.2 Spec case

  • One #[test] fn <snake_case_name>() per assertion.
  • File already exists (tests/spec/scalars.rs etc.) → add the test there. If the area has no file yet, create it under tests/spec/ and wire it into tests/spec/mod.rs.
  • Use noyalib::from_str (or the specific loader under test) directly; keep the assertion small — one input, one expected value.

4.3 Doctest

  • Every public item ships a # Examples block. CONTRIBUTING.md:116 makes this explicit.

  • For fallible APIs also add # Errors.

  • Errors-parse-a-repro pattern: a doctest that demonstrates an error variant must parse an actual reproducer YAML, not just construct the variant. Exemplar (src/error.rs:504–508):

    /// ```
    /// use noyalib::{Error, Value, from_str};
    /// let err = from_str::<Value>("1: a\n\"1\": b\n").unwrap_err();
    /// assert!(matches!(err, Error::KeyCollision(_)));
    /// ```
    

    Constructing Error::KeyCollision("1".into()) on its own is allowed only as a supplementary shape-example; the behaviour demonstration must parse.

4.4 Property test

  • Add to tests/proptest.rs (or a new properties_<area>.rs).
  • The universal property must be readable in one sentence (parse(emit(v)) == v, parse(format(s)) is total, etc.).
  • Shrink to minimal counterexamples; commit those as a fixed regression test in the nearest topical tests/*.rs file.

4.5 Fuzz-discovered regression

  • Add the minimised crashing input to tests/scanner_panic_regressions.rs (or the appropriate *_panic_regressions.rs) — one #[test] per input.
  • Keep the raw corpus entry under fuzz/corpus/<target>/ so the fuzzer will not re-discover it.

5. Acceptance thresholds

Before you claim "ready to merge":

Gate Threshold Where enforced
Official YAML Test Suite compliance >= 94.0 assertion; SKIP_LIST = &[] so total-loaded pass count must not drop tests/official_suite.rs:295–314 + review
Coverage — functions Target 98% / current gate 96% --fail-under-functions 96 (ci.yml:160 + shared-coverage.yml default)
Coverage — lines Target 98% / current gate 94% --fail-under-lines 94 (ci.yml:161 + shared-coverage.yml default)
Coverage — regions Target 98% / current gate 93% --fail-under-regions 93 (ci.yml:162 + shared-coverage.yml default)
Clippy -D warnings — zero warnings Workspace CI
Rustfmt Clean (make fmt) Workspace CI
Doctests All ~480 pass (cargo test --doc) Workspace CI
REUSE / SPDX Every source file carries a header REUSE workflow
Miri Focused pass green per PR Miri workflow
Fuzz smoke Differential fuzz 10s per target Differential fuzz workflow

The 98/98/98 target is the live campaign — see noyalib-coverage-campaign for the plan to lift the live gate (96/94/93 functions/lines/regions) up to 98/98/98. doc/TESTING.md still cites 95 / 93 / 92; that is documentation drift — trust the workflow files. A conformance regression (any yaml-test-suite case dropping from pass to fail/skip) is a release blocker, not a "next PR" item.


6. What does NOT count as evidence

Explicitly rejected. If any of the below is your entire justification, the PR is not ready:

  1. "It looks right." — reading the diff is not testing.
  2. A passing test with no failing-before counterpart. — the test must be shown (locally or via reviewer's mental replay of the diff) to fail on the pre-fix tree.
  3. Coverage-only tests that execute but do not assert. — every #[test] asserts something an engineer can read. A test that only calls from_str and drops the result is not a test; wrap it with assert!(matches!(...)) or assert_eq! on a value. The coverage_* family is not exempt from this rule.
  4. Bench deltas within noise. — Criterion prints a p-value; if the CI comparison harness (CodSpeed) does not flag the delta as significant, it is not a regression and it is not an improvement. Do not claim either.
  5. "Green on my machine." — full CI must be green. See the CI-always-green invariant.
  6. A doctest that only constructs a type. — for behavioural claims, parse a reproducer (see §4.3).

7. When NOT to use this skill

  • You need to run the tests or interpret llvm-cov output → noyalib-diagnostics-and-tooling.
  • You are working the 95 → 98 coverage push specifically → noyalib-coverage-campaign.
  • You are researching whether a bug was fixed before → noyalib-failure-archaeology.
  • You are deciding what to build → noyalib-architecture-contract.

Provenance

The read-only re-verification one-liners (test-file count, SPDX coverage, suite case count, the coded compliance >= floor, doctest count, wired coverage thresholds, fixture inventory) live in reference.md §R3. Run them before quoting any number here.

Date-stamp: 2026-07-07 · Crate version: v0.0.14 · Ground truth: crates/noyalib/tests/, doc/TESTING.md, CONTRIBUTING.md, Makefile.

04 同仓库更多 Skills

sebastienrousseau
sebastienrousseau/agtmls

example-skill

Use when the user needs this reusable workflow. Replace this with trigger-rich wording that says when to load the skill, what task it handles, and when not to use it.

★ 1
sebastienrousseau
sebastienrousseau/agtmls

writing-plans

Turn a vague or multi-step request into an ordered list of small steps, each with its own done-condition and verification. Load before starting work that touches several files, has unclear scope, or spans more than one session, and when a request needs decomposing before any code is written. Produces a plan a reviewer can check against, not a to-do list.

★ 1
sebastienrousseau
sebastienrousseau/agtmls

web-research-and-source-triage

Use when gathering current web evidence, comparing sources, triaging credibility, reading GitHub repositories, checking docs, or deciding whether a claim needs live verification.

★ 1
sebastienrousseau
sebastienrousseau/agtmls

vetting-a-skill-before-install

Use before installing, importing or enabling a third-party skill, plugin, marketplace or skills repository in Claude Code, Codex, Antigravity, Cursor or any agent. Pins the source to a commit, audits every skill and the agent configuration it ships, reads what the audit could not cover, and ends in install, mirror or refuse. Not for auditing this registry's own skills.

★ 1
sebastienrousseau
sebastienrousseau/agtmls

verification-before-completion

The gate between doing work and claiming it is done. Load before saying finished, fixed, working, or ready, before marking a task complete, and before reporting an outcome to a human. Converts each claim into the observation that would justify it, and names the rationalizations that let unverified work ship. Not the project's own evidence bar or test taxonomy.

★ 1
sebastienrousseau
sebastienrousseau/agtmls

using-agtmls

Meta-router for the AgtMLS skill catalog. Read first when unsure which AgtMLS skill applies, when several look relevant, when asking 'is there a skill for porting or noyalib work', or when onboarding to how the hub's skills and shared references fit together.

★ 1