[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M792RNM28BKD0R3FJ6SS4EAV":3,"skill-same:01M792RNM28BKD0R3FJ6SS4EAV":61},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":26,"commitSha":27,"skillDirectory":28,"forksCount":14,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":32,"ownerDescription":33,"updatedAt":34,"install":35,"resources":48},"01M792RNM28BKD0R3FJ6SS4EAV","experimentation",null,"Design A\u002FB tests, set sample size and duration, read results honestly, and run the backlog. Use when running A\u002FB split tests, sample sizing, or conversion experiments.","poorvith-mp\u002Fskills-marketing","poorvith-mp","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F277478855?v=4","01M78Z5K0YH1Z3PX815PGQ96ND","Poorvith M P","01M792P4C4Z12T0V4B97Z9MNX3",0,"MIT","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fexperimentation\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: experimentation\nlast_reviewed: 2026-09-06\ngroup: Measurement\ndescription: >-\n  Design A\u002FB tests, set sample size and duration, read results honestly, and run the backlog. Use\n  when running A\u002FB split tests, sample sizing, or conversion experiments.\n---\n\n# Experimentation\n\nGrowth experimentation is the scientific method applied to software conversion funnels. A\u002FB testing fails when teams launch underpowered tests, declare premature victory without statistical significance, or test trivial cosmetic changes. A professional experimentation practice requires statistical sample-size power calculations, structured hypothesis backlogs (ICE scoring), and disciplined holdout verification.\n\n## 1. Statistical Foundations & Test Sizing\nBefore launching an experiment, calculate mandatory sample size based on statistical parameters:\n- **Minimum Detectable Effect (MDE)**: The smallest relative conversion lift you care about detecting (e.g. 10% relative lift).\n- **Statistical Power ($1 - \\beta$)**: Standard baseline is **80%** (20% false negative rate).\n- **Statistical Significance ($\\alpha$)**: Standard baseline is **95%** ($p \u003C 0.05$, 5% false positive rate).\n- **Sample Size Formulation**:\n  $$n = \\frac{2 \\times (Z_{\\alpha\u002F2} + Z_{\\beta})^2 \\times p \\times (1 - p)}{\\delta^2}$$\n  - $p$: Baseline conversion rate; $\\delta$: Absolute detectable effect ($p \\times \\text{MDE}$).\n- *Runtime Constraint Rule*: If your monthly traffic requires more than 4 weeks to reach statistical power, the test is underpowered; choose a higher-traffic surface or test a bolder structural variation.\n\n## 2. The ICE Hypothesis Backlog Prioritization\nScore candidate experiments across three dimensions (1–10 scale):\n- **Impact**: How much will conversion increase if this hypothesis is correct? (10 = Core checkout page; 1 = Footer link).\n- **Confidence**: What empirical evidence supports this hypothesis? (10 = Directly backed by customer interview data and analytics drop-off; 1 = Pure founder intuition).\n- **Ease**: How quickly can engineering and design ship this test? (10 = Simple copy change in CMS; 1 = Requires new backend architecture).\n- **ICE Score**:\n  $$\\text{ICE Score} = \\frac{\\text{Impact} + \\text{Confidence} + \\text{Ease}}{3}$$\n\n## 3. The 4-Part Experiment Design Brief\nEvery test must be documented prior to deployment:\n1. **Observation & Data**: \"Analytics show 68% of users drop off at Step 2 of registration where phone number is requested.\"\n2. **Hypothesis**: \"Removing the optional phone number field will reduce perceived privacy friction and increase signup completion by 15%.\"\n3. **Primary Metric**: Signup completion rate (`signup_completed \u002F registration_page_views`).\n4. **Guardrail Metric**: Downstream activation rate (ensuring quality of signups does not degrade).\n\n## 4. Analyzing Test Results Honestly\n- **Never peek and stop early**: Checking a test daily and stopping when $p \u003C 0.05$ inflates false positive rates to >30%. Fix the sample size in advance and run the test for full business cycles (minimum 2 full weeks to smooth weekday\u002Fweekend variance).\n- **Inconclusive Results**: A flat test ($p > 0.05$) is not a failure; it proves the tested variable does not constrain conversion. Revert to control and archive the learning.\n\n## Critical Rules\n1. Never declare a winning variation without reaching 95% statistical significance and minimum pre-calculated sample size.\n2. Every experiment must run for at least two full 7-day calendar cycles to account for day-of-week seasonality.\n3. Every test must track a secondary guardrail metric to prevent optimizing for clicks at the expense of retention.\n\n## Verification Checklist\n- [ ] Sample size and test duration calculated prior to launching test variations.\n- [ ] Test instrumented to run across at least 2 full weekly business cycles.\n- [ ] Primary conversion event and secondary guardrail metrics established in analytics.\n- [ ] Experiment documented in central repository with hypothesis and ICE score.\n- [ ] Traffic split (50\u002F50) verified evenly distributed without session leakage.\n\n## Anti-Patterns\n- NEVER stop a test early because results look good on Day 3; early variance regression to the mean will wipe out illusory gains.\n- NEVER test minor button color shades when core value proposition copy remains unvalidated.\n- NEVER run multiple overlapping tests on the same user journey without isolating interaction effects.\n","# Experimentation\n\nGrowth experimentation is the scientific method applied to software conversion funnels. A\u002FB testing fails when teams launch underpowered tests, declare premature victory without statistical significance, or test trivial cosmetic changes. A professional experimentation practice requires statistical sample-size power calculations, structured hypothesis backlogs (ICE scoring), and disciplined holdout verification.\n\n## 1. Statistical Foundations & Test Sizing\nBefore launching an experiment, calculate mandatory sample size based on statistical parameters:\n- **Minimum Detectable Effect (MDE)**: The smallest relative conversion lift you care about detecting (e.g. 10% relative lift).\n- **Statistical Power ($1 - \\beta$)**: Standard baseline is **80%** (20% false negative rate).\n- **Statistical Significance ($\\alpha$)**: Standard baseline is **95%** ($p \u003C 0.05$, 5% false positive rate).\n- **Sample Size Formulation**:\n  $$n = \\frac{2 \\times (Z_{\\alpha\u002F2} + Z_{\\beta})^2 \\times p \\times (1 - p)}{\\delta^2}$$\n  - $p$: Baseline conversion rate; $\\delta$: Absolute detectable effect ($p \\times \\text{MDE}$).\n- *Runtime Constraint Rule*: If your monthly traffic requires more than 4 weeks to reach statistical power, the test is underpowered; choose a higher-traffic surface or test a bolder structural variation.\n\n## 2. The ICE Hypothesis Backlog Prioritization\nScore candidate experiments across three dimensions (1–10 scale):\n- **Impact**: How much will conversion increase if this hypothesis is correct? (10 = Core checkout page; 1 = Footer link).\n- **Confidence**: What empirical evidence supports this hypothesis? (10 = Directly backed by customer interview data and analytics drop-off; 1 = Pure founder intuition).\n- **Ease**: How quickly can engineering and design ship this test? (10 = Simple copy change in CMS; 1 = Requires new backend architecture).\n- **ICE Score**:\n  $$\\text{ICE Score} = \\frac{\\text{Impact} + \\text{Confidence} + \\text{Ease}}{3}$$\n\n## 3. The 4-Part Experiment Design Brief\nEvery test must be documented prior to deployment:\n1. **Observation & Data**: \"Analytics show 68% of users drop off at Step 2 of registration where phone number is requested.\"\n2. **Hypothesis**: \"Removing the optional phone number field will reduce perceived privacy friction and increase signup completion by 15%.\"\n3. **Primary Metric**: Signup completion rate (`signup_completed \u002F registration_page_views`).\n4. **Guardrail Metric**: Downstream activation rate (ensuring quality of signups does not degrade).\n\n## 4. Analyzing Test Results Honestly\n- **Never peek and stop early**: Checking a test daily and stopping when $p \u003C 0.05$ inflates false positive rates to >30%. Fix the sample size in advance and run the test for full business cycles (minimum 2 full weeks to smooth weekday\u002Fweekend variance).\n- **Inconclusive Results**: A flat test ($p > 0.05$) is not a failure; it proves the tested variable does not constrain conversion. Revert to control and archive the learning.\n\n## Critical Rules\n1. Never declare a winning variation without reaching 95% statistical significance and minimum pre-calculated sample size.\n2. Every experiment must run for at least two full 7-day calendar cycles to account for day-of-week seasonality.\n3. Every test must track a secondary guardrail metric to prevent optimizing for clicks at the expense of retention.\n\n## Verification Checklist\n- [ ] Sample size and test duration calculated prior to launching test variations.\n- [ ] Test instrumented to run across at least 2 full weekly business cycles.\n- [ ] Primary conversion event and secondary guardrail metrics established in analytics.\n- [ ] Experiment documented in central repository with hypothesis and ICE score.\n- [ ] Traffic split (50\u002F50) verified evenly distributed without session leakage.\n\n## Anti-Patterns\n- NEVER stop a test early because results look good on Day 3; early variance regression to the mean will wipe out illusory gains.\n- NEVER test minor button color shades when core value proposition copy remains unvalidated.\n- NEVER run multiple overlapping tests on the same user journey without isolating interaction effects.\n",{"name":5,"group":24,"description":7,"last_reviewed":25},"Measurement",1788652800000,"main","e9ea1069b8d051ec6f88c3bdc4290183b8e68c94","skills\u002Fexperimentation","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskillary","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp","poorvithmp.com","Shipping tools for problems people notice too late.","2026-10-02 21:32:44",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add poorvith-mp\u002Fskills-marketing --skill experimentation",{"tool":41,"command":42},"bunx","bunx skills add poorvith-mp\u002Fskills-marketing --skill experimentation",{"tool":44,"command":45},"pnpm","pnpm dlx skills add poorvith-mp\u002Fskills-marketing --skill experimentation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fexperimentation\u002FSKILL.md",true,[49,55],{"path":50,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":53,"sourceHtmlUrl":54},"evals\u002Frouting.yaml","REFERENCE","参考文档",1014,"https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fexperimentation\u002Fevals\u002Frouting.yaml",{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"experimentation.skill","OTHER","其他",2961,"https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fexperimentation\u002Fexperimentation.skill",[62,69,75,81,88,94],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M792Y3WSAZ5ABHX58QPMWXMH","youtube-clipper","Moved to modernized catalog in skillary. Removed in v4.1.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":19},"01M792Y1ABERS38QWQCJ9CKWJB","youtube-b-roll-maker","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-b-roll-maker\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M792XZCWKZV16GNK3QQR9M2X","x-twitter-intelligence-analyst","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fx-twitter-intelligence-analyst\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M792XTYRW8ZSHPSPVMK301QW","visual-assets","Generate on-brand images: OG cards, blog heroes, social graphics and mockups. Use when creating Open Graph cards, hero visuals, or brand graphics.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fvisual-assets\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M792XR5BZ510MFZETMPAAJBW","viral-hook-generator","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fviral-hook-generator\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M792XPJB02Q3HQTKKYZTKJ8A","video-script-writer","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-marketing\u002Fblob\u002Fmain\u002Fskills\u002Fvideo-script-writer\u002FSKILL.md",[],[]]