AGENT SKILL

evolution-engine

暂无简介

License 许可
MIT
所属仓库
zxpmail/ReqForge
最近更新
2026-10-07
Stars
★ 18
Forks
5

02 资源文件 4

03 SKILL.md 正文

<!-- forge: evolution-engine v1.0 -->

name: evolution-engine description: Auto-triggers on session init, or manually triggered when the user says "check if there are any rules to upgrade" or "check evolution suggestions". Called by the evolution-runner sub-agent. version: 1.0.0 updated: 2026-05-26 requires: []

<!-- begin: task --> [Task] Scan the accumulated data in ../../feedback/ and identify four types of evolution signals: 1. Rule graduation: feedback repeats 3+ times -> propose upgrading to an official rule 2. Skill optimization: feedback scores for a particular Skill are consistently low -> propose adjusting the Skill 3. New Skill proposal: an operation pattern keeps recurring but no Skill covers it -> propose creating a new Skill 4. Check graduation (escaped defect): a feedback entry has escape set (defect found past the gate that should have caught it) -> propose a PERMANENT check item. No 3-occurrence threshold — one escape is proof; waiting for 3 repeats means paying for 2 more escapes.

Signals found -> Generate proposals and return to the main Agent; execute after user confirmation.
No signals -> Return "no evolution suggestions".

<!-- end: task --> <!-- begin: not-for --> [Not For] - Recording individual feedback entries -> use /feedback-writer instead - Creating new Skills -> use /skill-builder instead - Fixing bugs or code issues -> use /bug-fixer or /code-review instead

<!-- end: not-for --> <!-- begin: dependency-check --> [Dependency Check] Automatically executed as the first step when the Skill starts.

Required:
- ../../feedback/FEEDBACK-INDEX.md → If missing, no feedback data exists; return "no evolution suggestions"
- At least one feedback file with occurrences >= 1 → If no feedback files exist, there is nothing to evolve
- .forge/editable-surface.json — Must exist and be readable. Defines directories/files the evolution engine is allowed to modify. If missing, create one with sensible defaults before proceeding.

Editable surface check (mandatory, before any proposal generation):
- Read .forge/editable-surface.json
- Verify that any proposed edit targets only paths listed under `editable.directories` or `editable.files`
- If a proposal would modify a path in `readonly`, reject it with reason: "path is outside editable surface"
- The editable-surface.json itself is in `readonly` — evolution engine cannot modify its own boundary definition

Optional:
- memory/ files → cross-reference task-history.md to validate whether pattern is real or coincidence

<!-- end: dependency-check --> <!-- begin: first-principles --> [First Principles] Data-Driven Evolution: No change without data. A single feedback entry is an anecdote, not a signal. Wait for the 3-occurrence threshold before proposing rule graduation. Let the data speak, not your intuition. Sole exception: escape-tagged entries (defect escaped past its catching gate) graduate immediately — the escape itself is expensive data, not an anecdote. Generator/Optimizer Recursion: The evolution engine is itself subject to evolution. The feedback-observer generates data (α), the evolution-engine optimizes rules (Ω). This cycle should recursively improve itself — the engine that proposes rule changes should also be evaluable and improvable through the same feedback loop. Minimum Lift: Prefer rule graduation (changing existing rules) over creating new Skills. A 3-line rule addition to an existing SKILL.md is faster to deploy and easier to maintain than a new Skill directory. Only propose new Skills when the pattern genuinely doesn't fit existing ones. Web-First: When proposing a new Skill or rule, WebSearch for existing best practices and community patterns. Don't invent from scratch what already has a well-known solution. Skill TDD Gate: Every proposal MUST include RED observation (what the Agent did without the rule), GREEN change (exact target file/section), Predicted effect, and Verify by. Proposals missing RED or Verify by are incomplete — do not present them to the user.

<!-- end: first-principles --> [Model Staleness Detection] When a feedback entry contains a model_version field that differs from the version used when the target Skill was written:

- Prioritize "rule outdated (model no longer needs this guidance)" over "rule needs strengthening"
- Check: is the same failure pattern still reproducible with the newer model? If not → propose rule retirement rather than rule graduation
- Record the model version delta in the proposal metadata

Rule retirement candidates:
- A rule that has ≥3 feedback entries, all from an older model version
- A rule that was graduated ≥2 versions ago and has 0 feedback entries from the current model
- Propose deactivation (not deletion) — keep the rule text in SKILL.md but mark as `status: deprecated` with the model version that made it obsolete

<!-- begin: proposal-quality-checklist --> [Proposal Quality Checklist] Before presenting an evolution proposal to the user, verify each criterion:

| Dimension | Must-Have | Recommended |
|-----------|-----------|-------------|
| **Data Support** | ≥3 feedback entries supporting the pattern | Cross-referenced against task-history.md to rule out false correlation |
| **Root Cause** | failure_class identified (skill-defect / execution-lapse / unset) | Skill score trend (Precision/Coverage) attached |
| **Proposed Change** | Specific SKILL.md section or rule text to add/change | Before/after diff example in proposal |
| **Verify-By** | Observable, time-bounded criterion to confirm fix | Suggested timeout for re-evaluation |
| **Priority** | RED (blocking) / GREEN (improvement) | Effort estimate (small/medium/large) |

Reference: see [First Principles] Skill TDD Gate for minimum proposal completeness.

<!-- end: proposal-quality-checklist --> <!-- begin: gotchas --> [Gotchas] Premature graduation: One feedback entry does not make a pattern. The 3-occurrence threshold exists for a reason — graduating too early means bloating the main control file (CLAUDE.md / AGENTS.md / reqforge.mdc) with one-off issues. Wait for the data. False correlation: "Skill X has low scores AND the user complained about Y" → these may be unrelated. Check if the feedback actually names Skill X before proposing changes to it. Ignoring sample size: A skill used 100 times with 80% satisfaction is fine. A skill used 5 times with 60% satisfaction is noise. Always check the denominator before acting on a score. Circular evolution: Rule A graduates from feedback, then generates more feedback, then graduates again as Rule A'. This is the ratchet spinning without progress. After graduating a rule, skip that pattern for N cycles.

<!-- end: gotchas --> <!-- begin: anti-rationalization-checklist --> [Held-In / Held-Out Validation] After applying a proposal (Skill edit, rule graduation, or hook change), validate the change:

Held-in split: Verify that the target failure pattern is resolved.
- Run the scenario that triggered the feedback (e.g., forge-verify on the same file that failed)
- Expected: PASS on the previously failing check
- If the check is forge-verify related, use: `node scripts/forge-verify/content-verify.mjs --from-config`

Held-out split: Verify that the change did not introduce regressions.
- Run forge-verify on files that previously passed and are not related to the target failure
- Expected: same verdicts as before the change
- Also check: a different skill or workflow that should be unaffected still works

Both splits must PASS before the proposal is considered applied.
If either fails, record the result but do NOT finalize the proposal. Revert and log.

Split configuration per proposal:
- `held_in_files`: files that should go from REJECT/UNCLEAR → PASS after the edit
- `held_out_files`: files that should remain PASS (regression check)

[Harness Search] When feedback suggests that the forge-verify configuration (thresholds, contracts, evidence requirements) is the bottleneck rather than skill text, propose a harness-level search:

- Baseline config lives at `.forge/content-verify.json`
- Candidate harnesses are stored in `.forge/harnesses/<name>/` with:
  - `config.json` — divergence threshold, runs, contracts, evidence_gates
  - `scores.json` — accuracy, llm_calls_saved, latency
  - `history.jsonl` — per-run results
- Generate a candidate by varying one dimension of the baseline config
  (e.g., tighten divergence threshold, add a contract pattern, switch evidence gate type)
- Evaluate candidate against held-in/held-out splits (same as [Held-In / Held-Out Validation])
- If candidate outperforms baseline on both splits, register it in `.forge/harnesses/index.json`
  and suggest it as a proposal

The `.forge/harnesses/` directory is in the editable surface (see `.forge/editable-surface.json`),
so evolution-engine can create new candidates. The production config at
`.forge/content-verify.json` is readonly — switching to a candidate requires human approval.

[Anti-Rationalization Checklist] → references/anti-rationalization.md 遇 premature graduation / skipping cross-reference / skipping Verify-by 时读取。

<!-- end: anti-rationalization-checklist --> <!-- begin: dimension-checklist --> [Dimension Checklist] See references/dimension-checklist.md for the full dimension checklist.

Must-have dimensions:
- **Data Support**: ≥3 feedback entries supporting the pattern
- **RED Observation**: specific Agent behavior cited
- **GREEN Change**: exact target file + section named
- **Verify-By**: observable, time-bounded criterion
- **Failure Classification**: skill-defect / execution-lapse / unset

<!-- end: dimension-checklist --> <!-- begin: quality-rubric --> [Quality Rubric] 8-item, 16-point scoring system. Ship threshold: ≥ 12 with no critical item scoring 0.

| # | Dimension | Pts | Critical | Scoring |
|---|-----------|-----|----------|---------|
| 1 | Signal accuracy | 2 | YES | 2 = ≥3 occurrences threshold respected, no premature graduation; 1 = Threshold applied but edge case slipped; 0 = Proposal based on single entry |
| 2 | RED observation quality | 2 | YES | 2 = Cites specific behavior with feedback excerpt; 1 = Generic description; 0 = Missing or "obvious problem" |
| 3 | GREEN change specificity | 2 | — | 2 = Exact target file + section + text to add/change; 1 = Target file named but section vague; 0 = "Update the Skill" |
| 4 | Verify-by criterion | 2 | YES | 2 = Observable, time-bounded, falsifiable; 1 = Vague ("should improve"); 0 = Missing |
| 5 | False correlation check | 2 | — | 2 = Cross-referenced task-history to validate pattern; 1 = Checked but conflicting data; 0 = No cross-reference |
| 6 | Denominator awareness | 2 | — | 2 = Usage count checked before acting on scores; 1 = Checked but ignored; 0 = Raw scores without context |
| 7 | WebSearch integration | 2 | — | 2 = Searched for best practices before proposing; 1 = Relevant known patterns used; 0 = Invented from scratch |
| 8 | Failure-class routing | 2 | — | 2 = skill-defect vs execution-lapse distinguished, proposal matches; 1 = Tagged but routing wrong; 0 = No routing |

**Scoring**: Run `pnpm validate-skill --score core/skills/evolution-engine` to compute.

<!-- end: quality-rubric --> <!-- begin: file-structure --> [File Structure] evolution-engine/ ├── SKILL.md # Main Skill definition (this file) └── references/ ├── anti-rationalization.md └── dimension-checklist.md

<!-- end: file-structure --> <!-- begin: output-style --> [Output Style] Tone: Scientific analyst reporting pattern signals — data-driven, conservative, actionable. Proposals are suggestions, not commands. Principles: - V Every proposal cites occurrence count + source — no data, no proposal - V A single feedback entry is an anecdote, not a signal - V Always check denominator (total skill uses) before acting on scores - X No premature graduation — wait for 3+ occurrences

<!-- end: output-style --> <!-- begin: output-artifacts --> [Output Artifacts] - Evolution proposals (screen output) — three types: Rule Graduation / Skill Optimization / New Skill. Each proposal includes confirm/skip options.

<!-- end: output-artifacts --> <!-- begin: workflow --> [Workflow]

Step 1: Scan Graduation Candidates
    Read ../../feedback/FEEDBACK-INDEX.md to locate all feedback files
    Read the frontmatter of each file
    Filter: occurrences >= 3 and graduated == false and skipped != true
    Escape override: entries with `escape` set are graduation candidates regardless of occurrence count (threshold exemption — see [Task] signal 4 and [Escape Routing])
    Determine graduation target:
    - source_skill is clear -> graduate to the corresponding SKILL.md
    - Involves multiple Skills or is global -> graduate to the main control file [General Rules]
    Cross-reference [Proposal Quality Checklist] Data Support dimension before proceeding.

    In addition to .claude/feedback/ files, also scan:
    - `.forge/trace/step-traces.jsonl` — step-level execution records from
      forge-loop runs. Look for entries where `attribution.failureClass` is set
      and the same failure pattern repeats across iterations. Treat repeated
      failure patterns (≥3 occurrences of same `step` + `failureClass`) as
      implicit feedback — they indicate Skill gaps exposed during automated
      execution.
    - When a feedback file exists AND step traces support the same pattern,
      cross-reference the trace's `attribution.reasoning` as additional evidence
      in the proposal's RED observation field.
    - If step traces reveal a failure pattern that has no corresponding feedback
      file, note this as "auto-detected from forge-loop traces" in the proposal.

Step 2: Check Skill Optimization Signals
    Scan scores fields in feedback/, grouped by source_skill
    Trigger conditions (any one met):
    - A Skill has the same dimension scored <= 2 for 3 consecutive times
    - A Skill's average score on some dimension over the last 5 entries <= 3
    - Total feedback occurrences for a Skill >= 5

Step 3: Check New Skill Signals
    Filter: occurrences >= 5 and not covered by any existing Skill
    -> Mark as "New Skill Candidate"

Step 4: Generate Proposals
    Signals found -> Generate structured proposals (see [Proposal Format])
    For each candidate: read linked feedback `failure_class` (if set) to choose routing (see [Failure-Class Routing])
    Drop any candidate missing **RED observation** or **Verify by** — log internally as "incomplete signal", do not show to user
    No signals -> Return "no evolution suggestions"

<!-- end: workflow --> <!-- begin: proposal-format --> [Proposal Format] Each proposal MUST include the Skill TDD quartet plus falsifiable verification (AHE-style — see agent-harness-seven-layer-map):

- **RED observation**: what the Agent did (or skipped) when the rule/skill was absent or ignored — cite feedback excerpt or session behavior, not intuition
- **GREEN change**: exact target (`SKILL.md` section, hook, `forge-bootstrap.md`, `CLAUDE.md` rule) and the text to add or change
- **Predicted effect**: which failure class should decrease (e.g. hallucinated paths, skipped tests, spec-before-code violations)
- **Verify by**: how to check after applying (e.g. "re-run scenario X", "zero feedback with tag `skill-defect:spec-before-code` in next 5 runs", `pnpm test`, `grep SKILL for new rule`)
- **Held-in files** (list): files that should go from REJECT/UNCLEAR → PASS after the edit — used after apply to confirm the fix
- **Held-out files** (list): files that should remain PASS — used after apply to confirm no regression
- **Model version** (string): the model version that will apply this proposal — used for future staleness detection

When forge-verify is available, held-in and held-out validation runs automatically after apply:
  `node scripts/forge-verify/content-verify.mjs --task "..." --files <held-in>`
  `node scripts/forge-verify/content-verify.mjs --task "..." --files <held-out>`

"**Evolution Suggestions** (N total)

 **Rule Graduation** (X items)
 1. [feedback title]: occurred [N] times (source: [source_skill]; failure_class: [skill-defect | execution-lapse | unset])
    Suggest writing to: [target file] at [target location]
    Summary: [one sentence]
    RED observation: [...]
    GREEN change: [...]
    Predicted effect: [...]
    Verify by: [...]
    -- Confirm / Skip

 **Skill Optimization** (X items)
 1. [Skill name]: [N] related feedback entries accumulated
    Optimization suggestion: [specific suggestion]
    RED observation: [...]
    GREEN change: [...]
    Predicted effect: [...]
    Verify by: [...]
    -- Confirm / Skip

 **New Skill Proposal** (X items)
 1. [operation pattern description]: occurred [N] times
    RED observation: [...]
    GREEN change: [create skill via skill-builder — outline]
    Predicted effect: [...]
    Verify by: [...]
    -- Confirm Create / Skip

 **Permanent Check** (X items)
 1. [escaped defect]: surfaced at [where], escaped [gate that should have caught it] (escape root cause: [gate missing / pattern miss / adversarial miss])
    RED observation: [...]
    GREEN change: [land per Escape Routing — machine check (hook / content-verify pattern / test) or checklist item, exact target file]
    Predicted effect: [...]
    Verify by: [...]
    -- Confirm / Skip"

<!-- end: proposal-format --> <!-- begin: bounded-skill-edits --> [Bounded Skill Edits] (SkillOpt-inspired — Skill Optimization proposals only) When GREEN change targets a user-project SKILL.md (not hooks / CLAUDE.md / bootstrap):

1. **At most 3** structured edits per proposal; each op is exactly one of: `add` | `delete` | `replace`
2. Each edit MUST state **which failure class** it fixes (link `failure_class` or RED observation)
3. **Verify by** MUST include held-out or regression check — e.g. `pnpm skill-eval <skill-name>` on `split: held-out` cases, or a named Phase scenario
4. If Verify by **fails** after user applied the edit → append to `.forge/skills/<name>/eval/rejected-edits.json` (do not re-propose the same edit without new evidence)
5. Before presenting, answer: (a) which failure? (b) harm to existing behavior? (c) measurable improvement on held-out?

Template: [evolution-bounded-edit-template.md](../../templates/evolution-bounded-edit-template.md) · Background: [skillopt-comparison.md](../../docs/skillopt-comparison.md)

<!-- end: bounded-skill-edits --> <!-- begin: failure-class-routing --> [Failure-Class Routing] Read failure_class from source feedback frontmatter when present:

| Tag | Meaning | Prefer changing |
|-----|---------|-----------------|
| `skill-defect` | Skill text missing, wrong, or outdated | Target `SKILL.md`, `references/`, or rule graduation |
| `execution-lapse` | Skill already covers behavior; Agent skipped or hooks failed | Bootstrap (`forge-bootstrap.md`), hook wiring, `CLAUDE.md` dispatch — **not** bloating Skill prose |
| `unset` | Unknown — infer from feedback body; if ambiguous, ask user before Confirm |

Mixed tags on one topic → split into two proposals if fixes differ (Skill vs hook).

<!-- end: failure-class-routing --> <!-- begin: escape-routing --> [Escape Routing] For escape-tagged entries ([Task] signal 4), classify the escape root cause to choose the landing form of the permanent check:

| Escape root cause | Permanent check lands as |
|-----|--------------------------|
| Gate missing entirely — no check covered this | New check item in the owning gate's checklist / SKILL.md (prose) |
| Gate existed, wording/pattern missed it | **Machine check**: content-verify pattern, grep hook, or test case |
| Check existed, adversarial case not covered | New negative test case / adversarial checklist question |

**Machine-first mandate**: anything expressible as grep/regex/test/hook MUST land as a machine check — pure prose is a last resort, not the default.
**Append-only**: permanent checks are only added, never silently removed or loosened. Retiring one goes through the Model Staleness path (deactivate, not delete) with an explicit proposal.

<!-- end: escape-routing --> <!-- begin: post-confirmation-execution --> [Post-Confirmation Execution] User confirms or skips each item: - Rule graduation -> Write feedback content into the target SKILL.md or main control file, mark graduated: true - Skill optimization -> Modify the corresponding SKILL.md - New Skill -> Invoke skill-builder to create - Permanent check -> Land per [Escape Routing] (machine check: hook script + hook registry row, content-verify pattern, or test file; prose: checklist entry), mark graduated: true - Skip -> Mark skipped: true, do not propose again

After apply: run the proposal's **Verify by** step and note pass/fail in the feedback topic or `memory/decisions-log.md`.

Held-in/held-out validation (when forge-verify is available):
- Run forge-verify on `held_in_files`: all must PASS. If any REJECT/UNCLEAR, the edit did not resolve the target failure.
- Run forge-verify on `held_out_files`: all must return the same verdict as before the edit. Any new REJECT = regression.
- Both splits must pass. Record the result with the model version in the proposal metadata for future staleness detection.
If `failure_class` was `execution-lapse` and fix was hook/bootstrap-only, do **not** duplicate the same rule inside multiple Skills.

<!-- end: post-confirmation-execution --> <!-- begin: yolo-mode --> [YOLO Mode] When FORGE_MODE=yolo, proposals are written to file instead of waiting for confirm/skip:

**Proposal Output** -> Write `changes/proposals.md`:
    Same fields as [Proposal Format] for every item: RED observation, GREEN change, Predicted effect, Verify by, failure_class when known.
    Omit proposals missing RED or Verify by. Skip per-item confirm/skip. Return to main Agent as:
    "N evolution proposals pending (see changes/proposals.md)"

<!-- end: yolo-mode --> <!-- begin: return-format --> [Return Format] Return to the main Agent: - Proposals exist: "N evolution suggestions pending" + full proposal content - No proposals: "No evolution suggestions"

<!-- end: return-format --> <!-- begin: initialization --> [Initialization] Step 1: Execute [Dependency Check] Step 2: Execute [Workflow]

<!-- end: initialization -->

04 同仓库更多 Skills