experiment-design-matrix
Use when planning experiments, ablations, or evaluation sweeps — before a costly run series, or when adding new cells (e.g. ablations) to a running campaign. Turns pre-registered hypotheses into a factor matrix with fair baselines, metrics, seeds, and stopping criteria. Not needed for trivial probes and smoke tests.
- License 许可
- MIT
- 最近更新
- 2026-09-05
- Stars
- ★ 4
- Forks
- 0
01 前言 Front Matter
- name
- experiment-design-matrix
- license
- MIT
- description
- Use when planning experiments, ablations, or evaluation sweeps — before a costly run series, or when adding new cells (e.g. ablations) to a running campaign. Turns pre-registered hypotheses into a factor matrix with fair baselines, metrics, seeds, and stopping criteria. Not needed for trivial probes and smoke tests.
03 SKILL.md 正文
Experiment Design Matrix
Design before looping runs. This skill receives the pre-registered claims
from hypothesis design (via .brief.md) and expands them into a concrete
factor matrix — so every later run is interpretable against what was
pre-committed.
Proportionality rule: the design record scales with the cost of the runs. A seconds-long probe or smoke test needs one sentence in chat; a sweep that burns dozens of runs needs the full matrix below. Never write a design doc heavier than the experiments it governs.
One design doc covers a whole campaign (a matrix of runs), not each individual run — per-run records are kept automatically by the experiment tools.
When to use
- Planning a new experiment, ablation study, or evaluation sweep
- Adding cells (ablations, controls, new levels) to a campaign already running
- Turning pre-registered hypotheses into a concrete factor matrix
- The user is about to fire ad-hoc scripts with no design record
Workflow
- Anchor —
research-brief-readto pick up the pre-registered claims, metrics, and thresholds this matrix serves (optional for small probes, never a gate). - Scaffold first —
experiment-loglist → create → scaffold under the workspace path; the design doc (design.md) is the island's first file. - Write the design doc —
- Question being answered, linked to the claim it tests
- Factors × levels (independent variables)
- Baselines and controls — and why each comparison is fair
- Metrics with precise definitions
- Seeds / repeats
- Stopping criteria
- Artifact paths that must survive
- Cheap-first order — smoke test → single matrix cell → full sweep.
- Confirm cost, then execute — before the full sweep, state the
estimate (runs × per-run time × resources) and confirm with the user via
question; smoke tests and single cells run without ceremony. Execute withexperiment-run(real artifact paths; batch runs by dependency; summarize with ```artifact fences when helpful). - Close the loop —
results-snapshotfor key results; killed cells stay recorded in the matrix — never silently drop unfavorable runs.
Done when
- Every factor has levels; every comparison has a fairness argument.
- Stopping criteria are written before the sweep starts.
- A smoke test has passed before any full sweep.
- The design doc lives in the island, next to the runs it governs.
Boundaries
- Sample size, power, and test selection →
statistical-rigor; this skill records what was decided, not the statistics behind it. - Runtime/environment details (shared
.prismnext/.venv, gates) live on the experiment tools — follow them, do not restate them here.
Rules
- Prefer fewer, interpretable runs over blind hyperparameter thrashing.
- When an ablation cannot answer the claim, redesign — do not only add seeds.
- Mid-campaign additions extend
design.mdin place; do not fork a second design doc for the same campaign.
04 同仓库更多 Skills
demo-multi-file
Local test skill with an extra notes file
writing-results
Use when drafting a Results / Experiments / Analysis section — narrating verified evidence from run receipts, with complete statistical reporting, derivation analysis, and honest treatment of negative results. Every number traces to a run; every claim survives the audit.
writing-related-work
Use when drafting a Related Work / literature review section or a standalone synthesis note — organizing prior work into a grounded, cited narrative or comparing methods across papers, from the project library plus staged external discovery. Never cites from memory.
writing-preliminaries
Use when drafting a Preliminaries / Background / Notation section — definitions, problem setup, and formal statements the rest of the paper builds on, with notation consistency and machine-checked formal claims. Reference patterns, not molds; include only what later sections actually use.
writing-methods
Use when drafting a Methods / Methodology / Approach / Model section — expounding the ideas and the reasoning behind the method, grounded in the project's brief, notes, and prior discussions, with empirical details from run receipts. Structure follows the idea's shape; reference angles, not fixed templates.
writing-introduction
Use when drafting or rewriting a paper's Introduction — framing the problem, positioning against prior work, and stating contributions as checkable claims, grounded in the brief, notes, and the library. Reference rhetorical patterns, adapted to the paper — never applied as molds.