evaluate-context-engineering

/evaluate:context-engineering

Audit a plugin, a single skill, or the always-loaded surface against the six context-engineering shifts Anthropic named for Claude 5 generation models, and report a C1–C6 card.

Two channels, deliberately kept apart:

Channel	What it is	Cost
M — mechanical	check-context-engineering.py, deterministic proxies over the tree	free
J — judgment	an isolated agent scoring the frozen 1–5 rubric with quoted evidence	one subagent, --judge only

Channel M ranks candidates; it cannot tell a load-bearing safety constraint from a restatement of a model default. Channel J makes that call. Never blend the two into one score — a mechanical proxy and a judgment answer different questions, and averaging them hides which one is driving a recommendation.

When to Use This Skill

Use this skill when...	Use alternative when...
Auditing whether a skill or rule is written for the old context rules	Checking structure/frontmatter/size lint → scripts/plugin-compliance-check.sh
A new Claude model shipped and the corpus needs re-reading	Measuring whether a skill still works → /evaluate:matrix
Deciding what to cut from CLAUDE.md and .claude/rules/	Actually migrating rules into skills → agent-patterns-plugin:meta-context-diet
You want ranked, evidence-backed cut candidates	Checking a skill reads clearly to a fresh agent → /evaluate:legibility

The six dimensions

Dim	Shift	What the scanner measures
C1	rules → judgment	Constraint markers per 1k chars, and how many sit next to a safety/irreversibility word
C2	examples → interface design	Fenced-block share of the body; whether args enumerates its alternatives
C3	upfront → progressive disclosure	Body size vs supporting files; disclosure levels (body → references → scripts)
C4	repetition → single source of truth	8-word shingle overlap between skills, rules, and CLAUDE.md
C5	(the >80% claim)	Always-loaded chars: CLAUDE.md + rules with no paths: scope; plus a per-rule size ceiling that applies to every rule, scoped or not
C6	simple specs → rich references	Multi-line shell blocks with no bundled script

Context

• Repo root: !git rev-parse --show-toplevel
• Plugins present: !find . -maxdepth 2 -name plugin.json -path '*/.claude-plugin/*'
• Always-loaded rules: !find . -maxdepth 3 -path '*/.claude/rules/*.md'

Parameters

Parse $ARGUMENTS:

Parameter	Default	Description
plugin	whole repo	Scope to one plugin directory, e.g. git-plugin
plugin/skill	—	Scope to one skill, e.g. git-plugin/skills/git-commit
--always-loaded	off	Score only CLAUDE.md and the unscoped rules
--judge	off	Also run Channel J on the target (one subagent per unit, max 3 units)
--budget N	100000	Always-loaded char budget for the C5 ratchet

Execution

Execute this audit:

Step 1: Run Channel M

python3 "${CLAUDE_SKILL_DIR}/scripts/check-context-engineering.py" --top 10

Add --target <plugin-or-skill-path> when the user scoped the run, and --always-loaded-budget <N> when they passed --budget.

When the user asks about the cost of a session that loads several repos (a portfolio checkout, sibling repos open together), add one --also <repo> per extra root plus --portfolio-budget <N>. Only the C5 surface is summed; C1–C4 and C6 stay scoped to --project-dir, since they measure authoring quality inside one marketplace. --also and --target are mutually exclusive.

Read the KEY=VALUE block. STATUS=ERROR means one of two things: an always-loaded surface is over budget — the primary repo's, or the portfolio's (C5_PORTFOLIO_*, with a per-repo C5_PORTFOLIO_BREAKDOWN) — or a single rule is over the per-rule ceiling (TYPE=rule_over_size_ceiling, C5_RULES_OVER_CEILING, --rule-size-ceiling, default 50000). The ceiling exists because the budget sums unscoped rules only, while a path-scoped rule still loads whole into every agent that touches a matching path; a 513 KB scoped rule once pushed a haiku subagent past its window unseen (#2667). ERRORs are always listed first, so --max-issues never hides the reason. Every other finding is a WARN candidate, not a verdict.

Step 2: Report the C1–C6 card

Render one row per dimension: the measured value, the count of flagged units, and the top offender. State the totals for C5_ALWAYS_LOADED_EST_TOKENS and SKILL_BODY_EST_TOKENS separately — they have different cost models and must not be summed.

Rank findings by always-loaded cost × frequency, never by raw size. An always-loaded rule costs on every turn; a skill body costs only when the skill fires; a references/ file or script costs only when read or run. A large skill body is a smaller problem than a medium-sized unscoped rule.

Step 3: Judge the top units (--judge only)

For at most 3 flagged units, spawn one isolated subagent each:

Task subagent_type: general-purpose
prompt: Read the frozen rubric at docs/benchmarks/2026-07-context-engineering/rubric.md
        and score <unit path> on C1-C6. Every score of 1, 2, 4 or 5 needs a
        verbatim quote from the unit, copied exactly. If you cannot quote for a
        score, score 3. Do not read the scanner's metrics.json.

Withhold Channel M's numbers from the judge — showing them anchors the judgment to the proxy the rubric is supposed to check independently. Run judges synchronously: the critique is the returned tool result (agent-patterns-plugin:cold-read-gate).

Step 4: Propose cuts, and one thing to keep

For each recommendation give: the unit, the dimension, the token cost, the proposed change, and whether the evidence is proven (a judged quote) or proxy-only (Channel M alone).

Include at least one unit that earns its tokens and should not be cut. A report with no counter-finding has an agreement bias, not a clean corpus — the scanner flags constraint density, and some of the densest text is the safety-critical text that should stay.

Step 5: Hand off, do not apply

Migrating rules into skills is agent-patterns-plugin:meta-context-diet's job — it carries the per-candidate confirmation gate for lossy edits to always-loaded files. This skill measures and recommends; it does not rewrite.

Agentic Optimizations

Context	Command
Whole-repo card	python3 "${CLAUDE_SKILL_DIR}/scripts/check-context-engineering.py"
One plugin	... --target git-plugin
Machine-readable	... --json
CI ratchet (exit 1 over budget)	... --strict
Portfolio budget across repos	... --also ../dotfiles --also ../repos-claude-config --portfolio-budget 155000
Full issue list	... --max-issues 0
Determinism check	... --json | sha256sum (must be stable across runs)

Related

• docs/benchmarks/2026-07-context-engineering/rubric.md — the frozen 1–5 anchors Channel J scores against
• .claude/rules/context-engineering.md — the standing house position and what it supersedes
• .claude/rules/skill-evaluation.md — the tiered cost model; ablation is the follow-up that turns a judgment into a measurement
• agent-patterns-plugin:meta-context-diet — migrates approved always-loaded candidates into skills