a-b-test-design
Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
- License 许可
- MIT
- 最近更新
- 2026-11-04
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- a-b-test-design
- description
- Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档agents/openai.yaml43 B
- 参考文档CREDITS.md345 B
- 许可证LICENSE1.0 KB
03 SKILL.md 正文
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome
04 同仓库更多 Skills
explainroo
Make an explainer video (MP4) with a voice-over using explainroo. The voice is made on this computer and no API key is needed. Use when the user asks for an explainer, how-to, walkthrough, launch or tutorial video, an animated diagram with a voice-over, or a short for YouTube, Reels or TikTok.
unslop
Cut AI tells from prose a human will read. Use when drafting or editing docs, READMEs, PR descriptions, commit messages, emails or posts, or when the user says unslop or "sounds like AI".
technical-writing
Layered technical-writing standard (Diátaxis, Google style, STE, Global English) for docs, RFCs, READMEs, PR descriptions and commit messages.
no-comments
Spawn Comment Sicko, fix accepted findings, and offer encodings for claimed constraints.
ponytail
Lazy senior dev mode: the smallest change that fully solves the task, and a reply a busy human understands in one read. Use on any coding task (writing, fixing, refactoring, reviewing, choosing dependencies) and when the user says "ponytail", "be lazy", "simplest solution", "yagni", or complains about over-engineering or bloat. Levels: lite, full (default), ultra.
ponytail-review
Quality review of a change: is the logic right, is it safe, does it hold under real load, is risky code tested, is it fast enough, and is every line needed. Reads the connected code, not only the diff. Each finding is explained in plain English. Use for "review this", "code review", "review the last commit", "review my PR", "is this over-engineered", /ponytail-review.