AGENT SKILL

experiment-design-matrix

Use when planning experiments, ablations, or evaluation sweeps — before a costly run series, or when adding new cells (e.g. ablations) to a running campaign. Turns pre-registered hypotheses into a factor matrix with fair baselines, metrics, seeds, and stopping criteria. Not needed for trivial probes and smoke tests.

License 许可
MIT
所属仓库
yibocat/prismnext
最近更新
2026-09-05
Stars
★ 4
Forks
0

01 前言 Front Matter

name
experiment-design-matrix
license
MIT
description
Use when planning experiments, ablations, or evaluation sweeps — before a costly run series, or when adding new cells (e.g. ablations) to a running campaign. Turns pre-registered hypotheses into a factor matrix with fair baselines, metrics, seeds, and stopping criteria. Not needed for trivial probes and smoke tests.

03 SKILL.md 正文

Experiment Design Matrix

Design before looping runs. This skill receives the pre-registered claims from hypothesis design (via .brief.md) and expands them into a concrete factor matrix — so every later run is interpretable against what was pre-committed.

Proportionality rule: the design record scales with the cost of the runs. A seconds-long probe or smoke test needs one sentence in chat; a sweep that burns dozens of runs needs the full matrix below. Never write a design doc heavier than the experiments it governs.

One design doc covers a whole campaign (a matrix of runs), not each individual run — per-run records are kept automatically by the experiment tools.

When to use

  • Planning a new experiment, ablation study, or evaluation sweep
  • Adding cells (ablations, controls, new levels) to a campaign already running
  • Turning pre-registered hypotheses into a concrete factor matrix
  • The user is about to fire ad-hoc scripts with no design record

Workflow

  1. Anchor — research-brief-read to pick up the pre-registered claims, metrics, and thresholds this matrix serves (optional for small probes, never a gate).
  2. Scaffold first — experiment-log list → create → scaffold under the workspace path; the design doc (design.md) is the island's first file.
  3. Write the design doc —
    • Question being answered, linked to the claim it tests
    • Factors × levels (independent variables)
    • Baselines and controls — and why each comparison is fair
    • Metrics with precise definitions
    • Seeds / repeats
    • Stopping criteria
    • Artifact paths that must survive
  4. Cheap-first order — smoke test → single matrix cell → full sweep.
  5. Confirm cost, then execute — before the full sweep, state the estimate (runs × per-run time × resources) and confirm with the user via question; smoke tests and single cells run without ceremony. Execute with experiment-run (real artifact paths; batch runs by dependency; summarize with ```artifact fences when helpful).
  6. Close the loop — results-snapshot for key results; killed cells stay recorded in the matrix — never silently drop unfavorable runs.

Done when

  • Every factor has levels; every comparison has a fairness argument.
  • Stopping criteria are written before the sweep starts.
  • A smoke test has passed before any full sweep.
  • The design doc lives in the island, next to the runs it governs.

Boundaries

  • Sample size, power, and test selection → statistical-rigor; this skill records what was decided, not the statistics behind it.
  • Runtime/environment details (shared .prismnext/.venv, gates) live on the experiment tools — follow them, do not restate them here.

Rules

  • Prefer fewer, interpretable runs over blind hyperparameter thrashing.
  • When an ablation cannot answer the claim, redesign — do not only add seeds.
  • Mid-campaign additions extend design.md in place; do not fork a second design doc for the same campaign.

04 同仓库更多 Skills

yibocat
yibocat/prismnext

demo-multi-file

Local test skill with an extra notes file

★ 4
yibocat
yibocat/prismnext

writing-results

Use when drafting a Results / Experiments / Analysis section — narrating verified evidence from run receipts, with complete statistical reporting, derivation analysis, and honest treatment of negative results. Every number traces to a run; every claim survives the audit.

★ 4
yibocat
yibocat/prismnext

writing-related-work

Use when drafting a Related Work / literature review section or a standalone synthesis note — organizing prior work into a grounded, cited narrative or comparing methods across papers, from the project library plus staged external discovery. Never cites from memory.

★ 4
yibocat
yibocat/prismnext

writing-preliminaries

Use when drafting a Preliminaries / Background / Notation section — definitions, problem setup, and formal statements the rest of the paper builds on, with notation consistency and machine-checked formal claims. Reference patterns, not molds; include only what later sections actually use.

★ 4
yibocat
yibocat/prismnext

writing-methods

Use when drafting a Methods / Methodology / Approach / Model section — expounding the ideas and the reasoning behind the method, grounded in the project's brief, notes, and prior discussions, with empirical details from run receipts. Structure follows the idea's shape; reference angles, not fixed templates.

★ 4
yibocat
yibocat/prismnext

writing-introduction

Use when drafting or rewriting a paper's Introduction — framing the problem, positioning against prior work, and stating contributions as checkable claims, grounded in the brief, notes, and the library. Reference rhetorical patterns, adapted to the paper — never applied as molds.

★ 4