aeon-skill-evals

aeon-skill-evals

Quality net for installed skills. Each skill can declare an assertion manifest; outputs are checked against it; failing assertions surface regressions and route concrete fixes.

Manifest format

token-movers:
  min_words: 200
  required_patterns: ["Top movers", "24h"]
  forbidden_patterns: ["I cannot", "as an AI"]
  must_cite_source: true
  min_distinct_items: 5

narrative-tracker:
  min_words: 400
  required_sections: ["TRANSITIONS", "POSITIONS", "MAP"]
  forbidden_patterns: ["exciting", "consider"]
  must_have_position_call: true

Supported assertions: min_words / max_words, required_patterns / forbidden_patterns, required_sections, must_cite_source, min_distinct_items, output_pattern (regex), and per-skill-family custom binary checks.

Operations

• eval — run every manifest-defined skill against its latest output.
• eval --skill=NAME — one skill.
• bootstrap --skill=NAME — generate a starter manifest from recent successful runs.

Regression states

State	Action
NEW_FAIL	Passing last run, failing now. Severity scales with pass streak.
NEW_PASS	Failing last run, passing now. Log the win.
CHRONIC	Failing > 3 consecutive runs. Recommend operator review.
STABLE_FAIL	Always failing. Manifest assertion mismatch — flag for review.

State in local evals-state.json.

Bootstrap mode

Samples last 5 successful runs of a skill. Computes:

• min_words at p25 of historical runs.
• Required patterns from common section headers.
• Forbidden patterns from default list (refusals, hedging filler).

Emits the proposed manifest for review. Never auto-commits — assertions need a human signoff.

Rules

• Assertions are observations, not specifications. Bootstrap before writing speculatively.
• Forbidden patterns catch hallucination markers and refusals. Keep the list tight; don't lint stylistic choices.
• Chronic failures get a recommendation, not a re-file.
• Manifest changes are reviewed; never auto-edited by this skill.