experimentation
Design A/B tests, set sample size and duration, read results honestly, and run the backlog. Use when running A/B split tests, sample sizing, or conversion experiments.
- License 许可
- MIT
- 最近更新
- 2026-10-02
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- experimentation
- group
- Measurement
- description
- Design A/B tests, set sample size and duration, read results honestly, and run the backlog. Use when running A/B split tests, sample sizing, or conversion experiments.
- last_reviewed
- 1788652800000
02 资源文件 2
- 参考文档evals/routing.yaml1014 B
- 其他experimentation.skill2.9 KB
03 SKILL.md 正文
Experimentation
Growth experimentation is the scientific method applied to software conversion funnels. A/B testing fails when teams launch underpowered tests, declare premature victory without statistical significance, or test trivial cosmetic changes. A professional experimentation practice requires statistical sample-size power calculations, structured hypothesis backlogs (ICE scoring), and disciplined holdout verification.
1. Statistical Foundations & Test Sizing
Before launching an experiment, calculate mandatory sample size based on statistical parameters:
- Minimum Detectable Effect (MDE): The smallest relative conversion lift you care about detecting (e.g. 10% relative lift).
- Statistical Power ($1 - \beta$): Standard baseline is 80% (20% false negative rate).
- Statistical Significance ($\alpha$): Standard baseline is 95% ($p < 0.05$, 5% false positive rate).
- Sample Size Formulation:
$$n = \frac{2 \times (Z_{\alpha/2} + Z_{\beta})^2 \times p \times (1 - p)}{\delta^2}$$
- $p$: Baseline conversion rate; $\delta$: Absolute detectable effect ($p \times \text{MDE}$).
- Runtime Constraint Rule: If your monthly traffic requires more than 4 weeks to reach statistical power, the test is underpowered; choose a higher-traffic surface or test a bolder structural variation.
2. The ICE Hypothesis Backlog Prioritization
Score candidate experiments across three dimensions (1–10 scale):
- Impact: How much will conversion increase if this hypothesis is correct? (10 = Core checkout page; 1 = Footer link).
- Confidence: What empirical evidence supports this hypothesis? (10 = Directly backed by customer interview data and analytics drop-off; 1 = Pure founder intuition).
- Ease: How quickly can engineering and design ship this test? (10 = Simple copy change in CMS; 1 = Requires new backend architecture).
- ICE Score: $$\text{ICE Score} = \frac{\text{Impact} + \text{Confidence} + \text{Ease}}{3}$$
3. The 4-Part Experiment Design Brief
Every test must be documented prior to deployment:
- Observation & Data: "Analytics show 68% of users drop off at Step 2 of registration where phone number is requested."
- Hypothesis: "Removing the optional phone number field will reduce perceived privacy friction and increase signup completion by 15%."
- Primary Metric: Signup completion rate (
signup_completed / registration_page_views). - Guardrail Metric: Downstream activation rate (ensuring quality of signups does not degrade).
4. Analyzing Test Results Honestly
- Never peek and stop early: Checking a test daily and stopping when $p < 0.05$ inflates false positive rates to >30%. Fix the sample size in advance and run the test for full business cycles (minimum 2 full weeks to smooth weekday/weekend variance).
- Inconclusive Results: A flat test ($p > 0.05$) is not a failure; it proves the tested variable does not constrain conversion. Revert to control and archive the learning.
Critical Rules
- Never declare a winning variation without reaching 95% statistical significance and minimum pre-calculated sample size.
- Every experiment must run for at least two full 7-day calendar cycles to account for day-of-week seasonality.
- Every test must track a secondary guardrail metric to prevent optimizing for clicks at the expense of retention.
Verification Checklist
- [ ] Sample size and test duration calculated prior to launching test variations.
- [ ] Test instrumented to run across at least 2 full weekly business cycles.
- [ ] Primary conversion event and secondary guardrail metrics established in analytics.
- [ ] Experiment documented in central repository with hypothesis and ICE score.
- [ ] Traffic split (50/50) verified evenly distributed without session leakage.
Anti-Patterns
- NEVER stop a test early because results look good on Day 3; early variance regression to the mean will wipe out illusory gains.
- NEVER test minor button color shades when core value proposition copy remains unvalidated.
- NEVER run multiple overlapping tests on the same user journey without isolating interaction effects.
04 同仓库更多 Skills
youtube-clipper
Moved to modernized catalog in skillary. Removed in v4.1.
youtube-b-roll-maker
Moved to modernized catalog in skillary. Removed in v4.1.
x-twitter-intelligence-analyst
Moved to modernized catalog in skillary. Removed in v4.1.
visual-assets
Generate on-brand images: OG cards, blog heroes, social graphics and mockups. Use when creating Open Graph cards, hero visuals, or brand graphics.
viral-hook-generator
Moved to modernized catalog in skillary. Removed in v4.1.
video-script-writer
Moved to modernized catalog in skillary. Removed in v4.1.