derailment-method
暂无简介
- License 许可
- MIT
- 最近更新
- 2026-10-26
- Stars
- ★ 0
- Forks
- 0
02 资源文件 6
- 参考文档references/evidence-and-anti-sycophancy.md4.7 KB
- 参考文档references/finding-schema.md4.0 KB
- 参考文档references/fix-patterns.md6.4 KB
- 参考文档references/reading-traces.md5.3 KB
- 参考文档references/root-cause-matrix.md4.5 KB
- 参考文档references/taxonomy.md6.9 KB
03 SKILL.md 正文
<!-- src: audit-skill-by-derailment/SKILL.md#Audit Skill by Derailment: The Native Two-Tier Auditor -->
Derailment method
A skill, a tool description, a --help page and an error message are a program that an LLM executes. When the agent drifts, the defect is in the text or affordance it was given. This method finds that text from the evidence of a real run and rewrites it. Two tiers do the work: a worker (a fresh agent that attempts a realistic task) and a reader (a read-only agent that studies the worker's whole execution). The orchestrator applies fixes and reruns.
<!-- src: audit-skill-by-derailment/SKILL.md#The 5 Non-Negotiable Invariants -->
The five invariants
- Fix the text, not the executor. Every remedy is an edit to the controlling text (a skill sentence, an example, a routing row, a tool description, an error message, a script) or to the tool itself. Never conclude "use a smarter agent". Rewrite or delete the sentence that misled; do not leave it and add a "do / don't" rule beside it.
- Claims are leads; the machine is the witness. An agent's closing report is a claim. Verify exit codes and what is on disk (
agentic-maker check receipts) before believing "done". See evidence-and-anti-sycophancy. - Never judge from a view that hides what the agent did. Read the harness's structured record (every tool call with full arguments, every result, hook events). A folded terminal view, a summary or the agent's own account is not evidence. Source ranking:
skills/derail/references/trace-sources.md. - Search by dimension; do not chase a quota. Cover trigger, discovery, syntax, thresholds, paths, resilience, mental model, recovery, completion and budget. "Nothing found in this dimension" is a valid answer; a list padded to a target length invites invented findings.
- No errata files. The corrected source is the deliverable. A line marked
[NICE]is load-bearing: never weaken or delete it without a finding that supersedes it.
Every finding names two things
- the controlling text: the exact sentence, example, row, help line or message that led the agent there - or its absence (
ABSENT) when nothing said what was needed; - what the agent believed because of it, and whether that belief was
observed(the trace shows reasoning) orinferred(reconstructed from what it did). Thinking text is usually not exposed, so most beliefs are inferred; say so.
Schema and handback rules: finding-schema.
<!-- src: audit-skill-by-derailment/SKILL.md#Severity & Root-Cause Cheat Sheet -->
Markers, severity, root cause (cheat sheet)
| Marker | Severity | Typical root cause | Fix family |
|---|---|---|---|
[STUCK] cannot continue |
P0 | S1 missing prerequisite, S2 contradiction, M2 unstated location | Prerequisite Surfacing, Path Reconciliation, Output Location |
[BROKE] command or selector threw |
P0 / P1 | O1 silent crash, O5 stale flag, O3 unhandled edge case | Error Recovery Addition, Option Modernization |
[GUESSED] invented an unstated decision |
P1 | M1 ambiguous threshold, M5 assumed knowledge | Threshold Concretization, Scaling Guidance |
| Premature completion, skipped gates | P1 | C1 completion illusion, C4 gate slack | Two-Tier Verification Enclosure, Rigid Phase Gating |
| Skipped destructive cleanup or mutation | P1 | C2 mutation hesitation | Staged Deprecation and Safe Neutralization |
Ran --help instead of a real probe |
P1 | C3 missing tooling awareness, M4 missing method | Actionable Pre-Flight Probe |
| Path drift or file recreation | P1 | S3 scattered information, M3 format inconsistency | Canonical Layout Enforcement |
| Target never chosen, or chosen wrongly | P1 | T1 not triggered, T2 collision, T3 over-triggered | Description rewrite at the trigger layer |
[NICE] the text prevented an error |
Keep | a load-bearing sentence or check | Never weaken or delete |
Three or more P1 findings in one phase collapse into a compound P0. The full code list (S1-S5, M1-M7, O1-O8, C1-C5, T1-T3) is in taxonomy; the matching fix for each code is in root-cause-matrix and fix-patterns.
Read when
| File | Read when |
|---|---|
| taxonomy | assigning a marker, severity and root-cause code |
| root-cause-matrix | choosing the fix for a code |
| fix-patterns | writing the rewrite itself |
| reading-traces | reading a trace: what drift looks like and where to look |
| evidence-and-anti-sycophancy | deciding whether "done" is true |
| finding-schema | producing or checking findings |
04 同仓库更多 Skills
design
暂无简介
targets
暂无简介
derail
Test a skill, plugin, CLI or MCP tool with fresh agents and fix the text that misled them. Started by you; spends usage.
build-skill
Create, redesign or merge a skill, grounded in facts, then hand over the test. Started by you.
build-mcp
Build, extend or repair an MCP server for agents, probe it, hand over the test. Started by you.
build-cli
Build or repair an agent-facing CLI, probe it, hand over the test. Started by you.