weekly-audit-patterns
The non-obvious invariants of the proactive nightly Claude audit workflow (.github/workflows/claude-weekly-audit.yml): the day-of-week mode selector, the stable dedup-key scheme that keeps it from re-filing findings every run, the four audit dimensions (security has its own nightly workflow, claude-security-audit.yml) and which one owns the Fail-Loudly check, and the `bug`-label → auto-fix promotion path. Read before editing that workflow, changing its cadence, changing how findings are filed/deduped, or adding an audit dimension.
- License 许可
- MIT
- 最近更新
- 2026-10-26
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- weekly-audit-patterns
- description
- The non-obvious invariants of the proactive nightly Claude audit workflow (.github/workflows/claude-weekly-audit.yml): the day-of-week mode selector, the stable dedup-key scheme that keeps it from re-filing findings every run, the four audit dimensions (security has its own nightly workflow, claude-security-audit.yml) and which one owns the Fail-Loudly check, and the `bug`-label → auto-fix promotion path. Read before editing that workflow, changing its cadence, changing how findings are filed/deduped, or adding an audit dimension.
03 SKILL.md 正文
Nightly Audit Patterns
.github/workflows/claude-weekly-audit.yml is the repo's one proactive Claude
lens — a scheduled deep review (not triggered by a PR) that fans out one read-only
Claude job per dimension and files a ranked triage issue. Everything else in
claude.yml is reactive. These are the invariants a future editor will otherwise break.
Naming: this ran weekly until it moved to a nightly cron (10:37 UTC ≈ 3am Pacific, deep on Sundays). Three identifiers deliberately kept the
weeklyname so the change did not orphan existing state or break references: the filename, theweekly-auditissue label (it is the dedup key — renaming it would re-file every open finding), and this skill'sname(referenced fromCLAUDE.md). Issue titles and the workflow display name did change, toNightly audit — ….
The four dimensions are mutually exclusive
correctness, docs, tests, features — a matrix of one Claude job each. Security
is NOT a dimension here — it moved to its own workflow, claude-security-audit.yml
(deterministic semgrep + a Claude taint/authz/suppression sweep, CVSS-scored, findings to
the private code-scanning tab). Don't re-add it here; that created two half-owners and the
single general "security lens" is what missed the hub tar-slip.
The lenses overlap unless the prompt keeps them disjoint, and the first run proved it:
correctness findings (a rollback that never rolls back, a poller returning null, a mode
that no-ops) leaked into features, and the priciest job's output vanished from the
triage issue. The decisive question for a broken thing: is the code wired but
misbehaving (correctness), never written (features), or contradicted by its docs
(docs)?
correctnessowns wired-but-broken behavior AND the CLAUDE.md "Fail Loudly" check (except Exception: pass, try/except returning a placeholder, silent degradation).featuresis only genuinely-missing/half-shipped capability — a TODO for code never written. Wired-but-broken is correctness, not features.docsowns doc-vs-code drift, including a feature documented as working but stubbed.testsin deep mode rolls plain "module X has no coverage" into ONE aggregate finding (dedup_key: tests:aggregate:untested-modules); separate findings only for risk-bearing untested logic (auth/gate/precedence/error-mapping/#1655).
Adding a dimension means: add it to the matrix, describe its disjoint lens in the shared
prompt, and the synthesis picks up its findings-<dim>.json automatically.
Published hub agents get the highest bar
Published agents are the shop window — the prompt makes every lens double-check them and
bump any gap up one severity (never 🟡; a default-path break is 🔴). Detect them by a
release_agent_<id>.yml, a shipped SCORECARD.md, or a released version: in
gaia-agent.yaml — currently only the email agent. The bar: in-sync high-quality
README/SPEC.md/SKILL.md/CHANGELOG.md (+ any contract spec) with a real eval SCORECARD.md
(gated by gaia.eval.scorecard_gate, never hand-authored) linked from the README; bulletproof
runtime code (no stubs/silent-fallbacks); solid #1655-grade tests. When a new agent publishes,
the detection generalizes to it automatically — no prompt edit needed.
Severity: 🔴 high · 🟠 medium · 🟡 low — no green
Green (🟢) reads as "pass/good," so it's banned. Broken behavior always outranks a missing test — never rate "module X has no tests" above a feature that's actually broken. High = security / data loss / default-path break; medium = broken user-facing behavior, a false doc, or a missing test guarding auth/a gate/destructive logic; low = missing tests on non-risk logic, cosmetic gaps. The synthesis emits a section per dimension (fixed order: Correctness, Features, Docs, Tests), grouping each finding under the dimension it declares — it never re-buckets.
Child issues are 🔴/🟠 only, and tagged auto-fixable
Only high/medium findings get a child issue (and thus one-click bug→auto-fix promotion).
🟡 (low) findings are listed in the parent triage issue and nowhere else — this caps
tracker churn (the first deep run filed 19 children, ~13 of them low-value coverage nits).
Each finding carries an auto_fixable boolean; the child body says whether applying bug
will let auto-fix land it (locatable/small) or whether it needs a human (a test suite, a
refactor) — so maintainers don't promote something auto-fix can't handle.
Findings are filtered in SYNTHESIS, never in the lenses
The lenses report everything they can ground in something they read — including
findings they are unsure about. Synthesis is the only filter. This split is deliberate
and load-bearing on Claude Opus 5: a lens prompt that says "be conservative", "precision
beats recall", or "skip anything you aren't confident is real" gets followed literally
— the model investigates just as hard and then silently reports less. Both this workflow
and claude-security-audit.yml previously said exactly that. Do not put it back. (The
security audit's own origin story is a recall failure: its predecessor missed the hub
tar-slip, CWE-22, by sampling and trusting a suppression comment.)
What each side owns:
- Lens: grounding only. Open the cited file, confirm the problem is present, confirm
it isn't already handled, and fill
evidencewith what you actually read. No evidence, no finding — the one hard filter upstream. Express doubt throughseverity(file it 🟡 and say so inwhy), never by dropping the item. - Synthesis: the real gate. Drops any finding at any severity whose evidence doesn't
substantiate its title, re-reads the cited
path/symbolfor every 🔴/🟠 before filing, demotes over-stated findings rather than deleting them, collapses an implausible flood on one area into a single finding, and does an intra-run cross-dimension dedup (a stubbed command flagged by bothdocsandcorrectnessis ONE issue, not two — keep the most severe, note the other lens).
If the tracker gets noisy, tighten synthesis. Do not re-muzzle the lenses.
Findings are written for a human, not an auditor
Every filed issue follows CLAUDE.md → How You
Communicate: the title and opening line say what
broke and who it hurts, in plain words; the path:line evidence goes underneath in a
sub-bullet or a <details> block. A finding that opens with a symbol name is one a
triager skips.
Dedup + suppression — the single biggest usability risk
Each finding carries dedup_key = <dimension>:<repo-relative-path>:<symbol-or-section>.
The symbol is a function/class name or doc heading — NEVER a line number (line numbers
move, so a line-based key re-files the same finding every run and the issue is unusable by
week 3). The key is embedded in each child body as <!-- audit-key: KEY -->. Synthesis
skips a finding whose key is in EITHER set:
- already-filed — keys on any open
weekly-auditissue (avoid duplicates). - suppressed-forever — keys on any
weekly-auditissue also labeledaudit-wontfix(open or closed). This is how you permanently silence accepted debt: close a child withaudit-wontfixand it never comes back. Without this, wontfix findings resurface every deep run forever.
Parent triage issues accumulate — the workflow NEVER closes one
Each run files a NEW parent (Nightly audit — <mode> — <run_id>; synthesis also matches the legacy Weekly audit — prefix so the chain survives the rename) and cross-links the
most recently created open parent with a comment ("Follow-up audit run filed as #N —
stays OPEN until its findings are addressed") — it does not close it. An earlier
version auto-closed the previous parent as "superseded," which silently hid an epic's
still-open child findings the moment the next run fired (#2010: 18 unaddressed children
went dark). Only a human maintainer may close a parent, and only after its findings
are addressed. Child issues stay open regardless (they're the actionable units) — that
was already true and is unchanged.
Since parents are never auto-closed, there can be several open Nightly audit — (and legacy Weekly audit —) issues at
once; synthesis always cross-links the highest-numbered (most recent) one, never assumes
there's "at most one." The tradeoff is deliberate: parents pile up until a maintainer
closes them, trading tracker tidiness for guaranteeing unaddressed findings stay visible.
The parent opens with a one-line tally (new/low/suppressed counts) for trend. On a
zero-new-findings run, synthesis posts nothing and does not touch any prior parent.
Security is out of scope here — it has its own workflow
Security moved to .github/workflows/claude-security-audit.yml (see its own patterns
skill). This audit files to a public triage issue, which is the wrong channel for a
vulnerability — so the prompt now tells every lens to hand off any security issue it
notices rather than file it, and the synthesis emits no security section. Do not
re-add a security dimension here or route a security finding into the public issue.
Promotion: bug label → existing auto-fix job
Child issues are opened without any auto-fix trigger label. A maintainer promotes
one to a PR by applying the bug label; the existing auto-fix job in
claude.yml (gated on label.name == 'bug' and contains(labels,'bug')) then
creates the branch + PR. There is no PR-creation code in this workflow — humans
gate every code change. A documentation/tests label alone does NOT trigger auto-fix;
route promotions through bug unless you deliberately widen the auto-fix if.
Cost & safety invariants
- Model is
AUDIT_MODEL(top-level env,claude-opus-5— ~half the token burn of Fable for comparable static-review quality, and the same $5/$25 as the Opus 4.8 it replaced). One place to change it; swap toclaude-fable-5for maximum depth at ~2x cost. A measured Fable deep run was ~$45 of API-equivalent subscription usage; Opus roughly halves that. ⚠️ Model support is gated by the pinnedclaude-code-actionversion — the action bundles the CLI that resolves model names, so a model newer than the pin is unresolvable and fails as a broken job rather than a clear error. Bump the SHA in the same change and prove it first withgh workflow run claude-auth-canary.yml -f model=<new-id>. - Serialized dimensions: the matrix runs
max-parallel: 1so the run is a steady drip, not a 4-job burst — this keeps it under the Max subscription's rolling (5-hour) rate limit. If you re-parallelize, expect a token spike that can trip that limit. - Skip-if-empty: normal mode exits in
preflightbefore any Claude call on a night with no commits. Deep mode never skips. This matters more now the cadence is nightly. - Cadence: nightly at 10:37 UTC (≈3am Pacific). The sibling
claude-security-audit.ymlalso runs nightly, deliberately ~1.4h earlier at 09:13 UTC so the two proactive Claude runs never stack on the shared subscription pool. - Modes:
normal= last N days' diff (window_days, default 1);deep= whole codebase, auto-selected on Sundays. ⚠️ The selector keys off ISO day-of-week (date -u +%u, 7 = Sunday), NOT day-of-month. It previously readday-of-month ≤ 7and was only correct because the cron fired on Mondays alone. Under the nightly cron that test matches the 1st–7th of every month — seven consecutive whole-codebase deep sweeps instead of one. Do not "simplify" it back to a day-of-month check. - Read-only: dimension + synthesis jobs run
--allowedTools Read,Grep,Glob,Bash— no Edit/Write, never install or run repo code (same rule asclaude.ymlreview jobs). - Concurrency group
claude-weekly-audit(not cancel-in-progress) so two scheduled runs never overlap and double-file. - Auth is the same OAuth-preferred / API-key-fallback wiring as every
claude.ymljob, covered byclaude-auth-canary.yml.
04 同仓库更多 Skills
web-search
Search the web via the Brave Search API. Use when the user asks about current events or external facts.
triage-support-ticket
Triage an inbound support ticket end to end. Use when the user pastes a support ticket or asks to triage one.
tool-mismatch
Declares a tool its tools.py never registers. Used to prove validation fails loudly with no partial load.
report-archive
Archive a generated report to the user's reports folder and register it with the team's report index. Use when the user asks to file, archive, or publish a finished report.
local-capability
Writes files on the local workspace. Used to prove Phase 1 refuses local-capability permissions.
incident-review
Run an incident postmortem. Stands in for a Claude Code skill library imported read-only from .claude/skills/.