improve-harness

Improve Harness

A periodic ritual that turns the last stretch of work into a better agent harness. It mines recent sessions for lessons, surveys every harness surface, researches what's gone stale, consolidates one sequenced plan, grills it, and converges the harness to a new canonical state — then merges that state across all your harness surfaces.

Run it like /improve-codebase-architecture: every week or two, or after a burst of heavy work has accumulated lessons and version drift. It is user-invoked — it only fires when you type it, because it mutates your live global harness.

The harness surfaces it touches

Every change lands in exactly one of these — name the target before editing:

• product repo — the repo you ship from. AGENTS.md/CLAUDE.md rule edits land here. Ships as a PR; if merging is a prod-deploy-class gate in your project, it needs explicit go-ahead.
• skills repo — the repo holding your vendored skills (cloned locally, e.g. ~/code/<your-skills-repo>); commit + push, installed via npx skills update -g. Not just skills — any standalone asset belongs here once it is genuinely reusable across projects and machines, e.g. a tooling/ script consumed by symlink into ~/.claude/scripts/. The test is reusability, not where you happened to author it and not whether that repo currently references it: "the skills repo has no copy to sync" answers is it out of sync, never should it live there. Measured 2026-08-02: that exact conflation filed a cross-project capability in the private harness repo, reachable from no other project, and took three human corrections to relocate.
• harness repo — your ~/.claude (itself a git repo, optionally backed by a private remote): agents, commands, hooks, global CLAUDE.md, settings, memory. Commit + push; never weaken its .gitignore.

Reversibility first

This ritual mutates your live global harness. Back up before any mutation — that is step 4(a), nothing destructive runs before it, and it runs before every mutation round, not once per session. Measured 2026-07-20: one session mutated the harness in three separate rounds and backed up before only the first; the second round shipped a compaction threshold four times tighter than the text it committed alongside it, live about two and a half hours, caught by an unrelated follow-up rather than by any gate. Every step is individually reversible (git revert, lock restore, plugin re-pin).

Process

Command-args may narrow the ask — one surface, one defect, one operation instead of the periodic sweep. A narrowed ask is a real branch of this ritual, not an exit from it.

• Say in the first reply that this is a scoped run, and name which of the five steps you are skipping and why. Silence here is how the whole ritual gets dropped: on 2026-07-15 a scoped run merged 41 files, reversed a three-day-old dated vendoring policy, and wrote nothing about itself, because the scoping decision quietly took steps 1, 3, 4b, 4d and 4f with it.
• Three things hold regardless of scope: step 4a (back up before every mutation round), step 5's merge gate, and step 4f (the run writes its own lessons down).
• A scoped run that would reverse a previously-recorded dated decision — a vendoring entry, a standing rule, a prior conclusion — routes that reversal through step 3 even when everything else is skipped. A policy pivot decided inside one clarifying answer is precisely the contradiction step 3 exists to catch.
• A scoped run still names the target surface before editing. On 2026-08-02 a scoped run landed genuinely cross-project tooling in one surface only, declared "Done", and took three human corrections to relocate it.

1. Launch the investigation — parallel dynamic Workflows

Fan the read-only investigation out into parallel background Workflows so they run while you do nothing. Run A, B, E and G every time; add C on a deeper/~quarterly pass (the agent-stack landscape moves slowly), and D whenever the period included a regression cluster, a prod incident, or an unstable-prod stretch (the production-stability pass).

Read the previous run's analytics/harness-improvement/<date>-program.md first and pass its SKIP_OR_WATCH, DID_NOT_SURVIVE and OPEN sections into every lane's prompt — B and C must not re-research a recorded negative, A must not re-mine a window a prior run already delta-scoped, and every OPEN item is re-derived by its own recorded command before this run decides anything. A run that starts from zero is a run that pays twice. Author the lanes from the templates — WORKFLOW.md is the index and the shared contract; each lane is its own file, so load only the ones you are firing:

• Workflow A — mine + consolidate (WORKFLOW-A.md). Session miners over the past week of BOTH runtimes — ~/.claude/projects/*/*.jsonl and ~/.codex/sessions/**/rollout-*.jsonl + history.jsonl — so friction lessons correlate across the whole harness, not just Claude. Seven survey lanes: GitHub merges, GitHub merge/CI friction, the issue tracker, worktrees + memory, a harness deep-audit, an instructions-file audit (per docs/instructions-files.md), and a self-audit of the previous run. A falsifier wave runs before consolidation. → one consolidated, sequenced plan with ranked new lessons and a concrete prune list.
  • A recurring friction loop is the highest-signal lesson there is — the same operation retried ≥3× to no effect. It is the canonical case for a hook (a settings.json PreToolUse/Stop guard that mechanically blocks the wrong move), not a doc edit. The motivating case: stuck on gh pr merge four times before learning "resolve all PR review comments before merging" — that friction should have surfaced itself as a hook proposal, never waited on the human to notice it.
  • Lesson↔rule correlation feeds the doc edits. A mined lesson an existing rule should have prevented strengthens that rule or promotes it to a hook; a lesson with no rule home proposes the addition at the right layer; a rule whose subject no longer exists gets pruned.
• Workflow B — vendor delta: news, changelogs, docs + models (WORKFLOW-B.md). What the vendors actually shipped since the last run — CLI release deltas, agent/skill spec changes, model lineups + deprecations, published agent-design guidance, the second runtime's CLI, adjacent agent platforms — each mapped to a concrete harness edit. Every claim carries a dated primary source; config keys, env vars and flags are corroborated against the installed binary. Includes the pin-vs-enforcement check. → a version matrix, an upgrade plan, and a skip_or_watch list.
• Workflow C — agent-stack landscape (periodic) (WORKFLOW-C.md). Live-verify the best/newest GitHub agent stacks and gap-analyze our owned agents against them. → stay/cherry-pick/adopt. The standing answer is own + cherry-pick specific roles, never adopt a stack wholesale — a competing framework violates one-architecture (see GOTCHAS.md).
• Workflow D — production-stability / regression RCA (conditional) (WORKFLOW-D.md). One agent RCAs each incident grounded in its diff and deploy-timestamp-vs-onset; findings are liveness-checked at HEAD; classes get a guardrail each; a separate skeptic adversarially REFUTES every guardrail (3 of 5 ship as false-security otherwise). → a prioritized program spanning documentation, CI gates, harness rules, cross-session coordination and standing invariants. It operationalizes the load-bearing finding that local green is structurally blind to whole defect classes, so the lever is a cheap standing invariant at the un-mockable layer — never "understand the feature better".
• Workflow G — vendor-surface adoption + interaction quality (every run) (WORKFLOW-G.md). B asks what the vendor shipped; G asks what the vendor already offers that we never adopted, and whether the surfaces we did adopt are working. Measured 2026-08-09: the operator's loudest standing complaint — replies too long and hard to follow up — had a vendor surface sitting entirely unused (~/.claude/output-styles/ absent, outputStyle zero hits in settings) while the same rule had been written into CLAUDE.md twice and reversed once. Nothing had shipped, so B could not have caught it. G also owns the standing reply-shape grade from ~/.claude/analytics/reply-shape.jsonl, scoring VOLUME and FOLLOWABILITY as separate numbers against the 2026-08-09 baseline (median 2,270 chars; 26.4% of final turns with ≥3 headers versus 3.8% posing ≥3 questions — turns end multi-topic with no question). → a dated report plus one appended row under analytics/harness-adoption/, with every recommendation carrying a tier (mechanism > mechanism-carrying-prose > subtraction > prose) and a falsifiable metric. A defect that already has a prose rule may not receive another prose rule — the answer is a tier change or an instrument, never a third wording.
• Workflow E — context & compaction economics (every run) (WORKFLOW-E.md). The other lanes ask "is the harness correct?"; E asks "what does it cost, and is that improving?" Measures context per turn, the fixed preamble floor, payload composition, compaction peak-before-reset, subagent peaks and model mix — then audits the compaction knobs, composing the effective threshold from ALL of them together and reconciling it against measured behavior. → a dated report plus one appended row in the persistent trend ledger under the harness repo's analytics/context-economy/. The trend is the product, not the snapshot: each run grades the last one, and a change whose target metric didn't move gets named a failed intervention instead of being quietly re-tuned.

All run read-only. Completion criterion: every launched Workflow has returned its structured plan; nothing in the harness has been mutated yet.

2. Reconcile into one sequenced program

Merge the plans into a single ordered program: prune → vendor → upgrade → doc-edits → guardrails+hooks → context-economy ledger → memory. (Workflow E's and G's recommendations route by their own surface field; both reports + trend rows always land, even when no knob and no surface changes. G's tier ordering survives the merge — when G and another lane propose competing fixes for one defect, the higher-tier fix wins, and a prose item aimed at a defect that already has a prose rule is dropped here rather than carried into step 3.) (When D ran, weave its prioritized robustness program in as the guardrails step — CI gates, standing invariants, cross-session tooling, doc-rules — each routed to its surface; drop any guardrail D's adversarial verifier rated insufficient unless its refinement is applied.) Every recurring-friction lesson from Workflow A's friction lane gets a routing decision here: a settings.json hook (mechanical block — highest leverage), a standing CLAUDE.md/AGENTS.md rule (prose guard), or both. Prefer a hook whenever the wrong move is mechanically detectable at a tool boundary (e.g. a PreToolUse guard on gh pr merge that refuses while review threads are unresolved) — a rule the model "should remember" is weaker than a hook the harness enforces. Reconcile overlaps — the model audit is authoritative over the audit's coarse "re-point everything" (bump only genuinely-stale IDs); a delete-target needs no model bump. Adopt the recommended defaults for any low-stakes open question; surface only genuine risk-class forks (e.g. a MAJOR plugin bump). Completion criterion: the program is written to the harness repo at analytics/harness-improvement/<YYYY-MM-DD>-program.md — every item carrying a target surface, a risk rating, and its verifying probe — with three standing sections that the NEXT run reads before it dispatches anything: SKIP_OR_WATCH (refuted, already-adopted and app-runtime-class items, one line each with the reason), DID_NOT_SURVIVE (killed items with the probe that killed them), and OPEN (items deferred, blocked, or applied-but-unbaked, each with the command that re-derives its status now). and the program has passed tooling/workflow/lane_coverage_check.py against the run's journal.jsonl (it fails naming any dispatched lane the program does not cite, or any lane that died — see WORKFLOW.md). A plan that lives only in the conversation is not an artifact — it is gone at the next compaction, and the next run pays to rediscover it. Measured 2026-08-02: a run re-mined a window a run three days earlier had already delta-scoped, 11 agents and ~1.22M tokens, and separately found a settings pin from an earlier run's plan that was never applied and that nothing was tracking.

3. Grill the plan

Fire Workflow F — plan falsifier (WORKFLOW-F.md) on the consolidated plan before mutating anything, one lane per plan item, and run /grill-with-docs on whatever survives if the plan contains a genuine design fork. A self-narrated inline paraphrase — "grilling it against your own rules surfaced..." — is not this step; grade it by artifact the same way a dispatched lane is graded. Measured across 13 runs: this step has been narrated and skipped at least eight times, and the 2026-07-06 plan that self-certified shipped a hook that blocked autonomous work for eight hours. Completion criterion: a Workflow F run id for this run, and every killed item recorded with its probe.

4. Execute, in order

Each phase has traps — read GOTCHAS.md before running it.

• a. Back up the harness (lean tar of touched dirs + ~/.agents + the skill-lock; ~/.claude is itself a git repo). Verify the backup contains the files you're about to delete.
• b. Prune dead config — verify every delete target exists first, then remove and confirm no dangling references remain.
• c. Vendor — fork or sync the skills repo, promote/adapt skills, then cut the global install over to the fork (npx skills add <fork> -g -y -s <name> per skill). Confirm every lock row points at the fork and no symlink dangles.
• d. Upgrade — claude plugin marketplace update then claude plugin update per plugin; CLIs; bump only genuinely-stale model pins; prune dangling marketplaces. Plugin updates need a restart to apply; flag MAJOR bumps for a bake you can't do mid-session.
• e. Doc-edits + hooks — apply the rule changes to the product repo's AGENTS.md (for the PR) and the global CLAUDE.md (harness repo); apply approved hooks to ~/.claude/settings.json (+ a script under ~/.claude/hooks/ if non-trivial) — harness repo. Instructions-file edits follow docs/instructions-files.md: additions come from the lesson↔rule correlation, prunes/demotions from the instructions audit, and the AGENTS.md↔CLAUDE.md pair stays one-canonical-plus-pointer; note instructions files load at session start (running sessions keep the old contract until restart), unlike skills, which load at invocation. A hook can block a tool call, so treat it like a guardrail: replay it against the ACTUAL transcripts that motivated it — it must fire at the real friction point, stay silent on a healthy control session and on the current session, and emit a bounded number of times over a long incident. A dry-run against fixtures you wrote yourself proves only that the hook matches your model of the bug (observed: a guard that passed 9/9 invented cases was a false negative on 3 of 3 real incidents and blocked live work). Derive the trigger from measuring a candidate signal across the incidents, not from the lesson's prose. Confirm the event's output contract against the installed binary, and never weaken ~/.claude's .gitignore to land a hook script (see GOTCHAS.md).
• e2. Analytics ledgers (E and G) — commit both dated reports + both appended TRENDS.md rows to the harness repo, under analytics/context-economy/ and analytics/harness-adoption/ (they are the run's baseline; without the commit the next run has nothing to diff against). A run where nothing changed still writes its rows — an unchanged number is the evidence that grades the last run's intervention. Apply G's recommendations by tier, adopting the instrument before the behavior change wherever G marked one owed. Apply E's ranked recommendations to their named surfaces — settings knobs, instructions-file rules, a hook, or a plugin/MCP roster prune. Change ONE compaction knob at a time and record the expected effect in the ledger row, so the next run can grade it; two knobs moved together are unattributable, and a knob that multiplies with another can undo the intent silently.
• f. Memory + skill-learning promotion — run the memory-consolidation pass first (keep the index lean), then write the new lesson files. Then promote skill learnings by scope (see docs/skill-memory.md): a learning that recurs ≥3× in a skill's private overlay (~/.claude/skills-overlay/<skill>/LEARNINGS.md) or across project memories gets de-identified and promoted into that skill's public seed LEARNINGS.md via the skills-repo PR — promotion is the ONLY way the seed grows. Keep secrets / customer / project specifics OUT of the seed (they stay in the overlay or project memory).

Completion criterion: every planned item is applied and locally verified; the only outstanding work is the human-gated restart + bake.

5. Merge everything, verify clean

Land the work across all your harness surfaces: the product-repo AGENTS.md PR (if merge-to-main is a prod deploy in your project, do not merge without an explicit go-ahead), the skills repo push, and the harness repo commit + push. Then verify each repo is clean and local == remote — and that every new artifact actually PROPAGATED to the runtime, which a clean merge does NOT prove. Skills self-wire via the repo's wiring tool on the post-merge hook; a new asset kind (agents, commands) may have no such path and land as a silent no-op that reads as success — resolve each new artifact's symlink into the canonical clone, sweep for dangling links, and confirm the count matches the source (see GOTCHAS.md). Extend the wiring tool rather than hand-linking; a hand-link fixes one machine once and is forgotten by the next run. Completion criterion: all surfaces clean, every new artifact verified reachable from its runtime root, the version matrix reflecting the new canonical state, this run's program written to analytics/harness-improvement/<YYYY-MM-DD>-program.md, and — for any run that reached step 4 — an overlay entry dated today, confirmed by grep -q "^## $(date +%Y-%m-%d)" ~/.claude/skills-overlay/improve-harness/LEARNINGS.md. A run that mutates the harness and writes zero bytes about itself is incomplete, the same tier as an unmerged surface: six or seven of about eleven runs through 2026-08-02 left no trace at all, which is why the same defects kept being rediscovered from scratch. The only remaining items are the restart and the bake handed to the human.