[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GXB6PAN1AE612GSWZPW5GV":3,"skill-same:01M6GXB6PAN1AE612GSWZPW5GV":70},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":28,"linkOnly":29,"markdownContent":30,"markdownBody":31,"frontMatter":32,"refName":33,"commitSha":34,"skillDirectory":35,"forksCount":36,"repositoryHtmlUrl":37,"repositoryHomepageUrl":6,"ownerHtmlUrl":38,"ownerBlogUrl":39,"ownerDescription":40,"updatedAt":41,"install":42,"resources":55},"01M6GXB6PAN1AE612GSWZPW5GV","adjust-for-multiple-comparisons",null,"Use when many metrics or slices are tested at once. Applies a multiplicity correction before calling any one of them significant.","jiayaoqijia\u002Fcryptoskill","jiayaoqijia","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1453879?v=4","01M6GMYTX7Q7S504VC6E756ENR","YQ","01M6GMYV1QKJ1CC75MTTGT9PR0",81,"MIT","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-adjust-for-multiple-comparisons\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-adjust-for-multiple-comparisons\u002FSKILL.md",[],[20,21,22,23,24,25,26,27],"adjust","adjust for multiple comparisons","for","metrics","use","calling","at","when","2026-11-10 10:12:30",false,"---\nname: adjust-for-multiple-comparisons\ndescription: Use when many metrics or slices are tested at once. Applies a multiplicity correction before calling any one of them significant.\n---\n\n# Adjust for Multiple Comparisons\n\nTest 20 metrics at alpha 0.05 and about one will look significant by chance. If you did not bound the family of tests in advance, the headline is a coin flip wearing a lab coat.\n\n## Procedure\n\n1. Count the hypotheses actually tested: primary, secondary, and every slice a human will look at.\n2. Pre-define the family, then apply a correction. Benjamini-Hochberg for exploration (control the false discovery rate), Bonferroni for confirmation (control the family-wise error):\n   ```python\n   from statsmodels.stats.multitest import multipletests\n   p = [0.031, 0.008, 0.21, 0.044, 0.002, 0.12]      # six metrics\n   rej, padj, _, _ = multipletests(p, alpha=0.05, method='fdr_bh')\n   print(list(zip(p, [round(x,4) for x in padj], rej)))\n   ```\n3. Report adjusted p-values (or q-values) in the table, and mark which passed after correction.\n4. Use Bonferroni when one false positive is costly and the family is small; use FDR when scanning many metrics to prioritise.\n5. Correlated metrics (clicks and CTR) are not independent tests; note this when interpreting a corrected result.\n6. Never add a slice to the confirmatory family after seeing it move; that slice belongs in exploratory.\n\n## Pitfalls\n\n- Guardrail metrics are tests too; a set of ten guardrails needs the same correction.\n- Picking the one significant segment out of twelve and reporting it as the finding is the failure this prevents.\n- Bonferroni on a large scanning family is too conservative and hides real signals; match the method to the intent.\n- Correcting only the printed metrics while silently checking dozens more does not fix the multiplicity.\n- FDR controls the expected proportion of false discoveries, not the chance of any — say which guarantee you are giving.\n- Pre-specify the correction before results are seen; choosing Bonferroni or FDR afterwards is another degree of freedom.\n- Reusing one control against several arms in an A\u002FB\u002Fn test is multiplicity too — split alpha across the comparisons.\n\n## Verification\n\n    python3 -c \"from statsmodels.stats.multitest import multipletests as m; r=m([.01,.02,.03,.2,.4],method='fdr_bh'); print(r[0], r[1].round(3))\"\n\nReport: \"31 metrics tested; after BH-FDR at 0.05, 2 remain significant (primary q=0.004, latency guardrail q=0.031). The other 29, including two raw p\u003C0.05 slice wins, are exploratory.\"\n","# Adjust for Multiple Comparisons\n\nTest 20 metrics at alpha 0.05 and about one will look significant by chance. If you did not bound the family of tests in advance, the headline is a coin flip wearing a lab coat.\n\n## Procedure\n\n1. Count the hypotheses actually tested: primary, secondary, and every slice a human will look at.\n2. Pre-define the family, then apply a correction. Benjamini-Hochberg for exploration (control the false discovery rate), Bonferroni for confirmation (control the family-wise error):\n   ```python\n   from statsmodels.stats.multitest import multipletests\n   p = [0.031, 0.008, 0.21, 0.044, 0.002, 0.12]      # six metrics\n   rej, padj, _, _ = multipletests(p, alpha=0.05, method='fdr_bh')\n   print(list(zip(p, [round(x,4) for x in padj], rej)))\n   ```\n3. Report adjusted p-values (or q-values) in the table, and mark which passed after correction.\n4. Use Bonferroni when one false positive is costly and the family is small; use FDR when scanning many metrics to prioritise.\n5. Correlated metrics (clicks and CTR) are not independent tests; note this when interpreting a corrected result.\n6. Never add a slice to the confirmatory family after seeing it move; that slice belongs in exploratory.\n\n## Pitfalls\n\n- Guardrail metrics are tests too; a set of ten guardrails needs the same correction.\n- Picking the one significant segment out of twelve and reporting it as the finding is the failure this prevents.\n- Bonferroni on a large scanning family is too conservative and hides real signals; match the method to the intent.\n- Correcting only the printed metrics while silently checking dozens more does not fix the multiplicity.\n- FDR controls the expected proportion of false discoveries, not the chance of any — say which guarantee you are giving.\n- Pre-specify the correction before results are seen; choosing Bonferroni or FDR afterwards is another degree of freedom.\n- Reusing one control against several arms in an A\u002FB\u002Fn test is multiplicity too — split alpha across the comparisons.\n\n## Verification\n\n    python3 -c \"from statsmodels.stats.multitest import multipletests as m; r=m([.01,.02,.03,.2,.4],method='fdr_bh'); print(r[0], r[1].round(3))\"\n\nReport: \"31 metrics tested; after BH-FDR at 0.05, 2 remain significant (primary q=0.004, latency guardrail q=0.031). The other 29, including two raw p\u003C0.05 slice wins, are exploratory.\"\n",{"name":5,"description":7},"main","3df1d0d67b30ff15236561f8c0f9f06d33f42dfd","skills\u002Fpayments\u002F0xklen-adjust-for-multiple-comparisons",17,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia","https:\u002F\u002Feth2030.com","Hardcore infra. Building @alt-research and applied cryptography.","2026-11-14 02:27:51",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add jiayaoqijia\u002Fcryptoskill --skill adjust-for-multiple-comparisons",{"tool":48,"command":49},"bunx","bunx skills add jiayaoqijia\u002Fcryptoskill --skill adjust-for-multiple-comparisons",{"tool":51,"command":52},"pnpm","pnpm dlx skills add jiayaoqijia\u002Fcryptoskill --skill adjust-for-multiple-comparisons","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-adjust-for-multiple-comparisons\u002FSKILL.md",true,[56,60,66],{"path":57,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":16},"LICENSE","许可证",1063,{"path":61,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":64,"sourceHtmlUrl":65},"SOURCE.md","REFERENCE","参考文档",357,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-adjust-for-multiple-comparisons\u002FSOURCE.md",{"path":67,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":68,"sourceHtmlUrl":69},"TRUST.auto.yaml",2217,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-adjust-for-multiple-comparisons\u002FTRUST.auto.yaml",[71,99,121,139,152,162],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":75,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":28},"01M712C436EQYE43342Q4E7GG1","recipe-performance","Diagnose a specific MetaMask user flow's real-app performance and Sentry critical-user-flow telemetry with deterministic mm-harness recipes. Use for CUF boundaries, cold\u002Fwarm\u002Fbackground-resume behavior, latency regressions, lifecycle tags, duplicate or blocking network requests, long tasks, layout\u002Fpaint cost, screen redraws, React rerenders, or browser trace analysis where tests alone cannot prove the real app.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FSKILL.md",[],[79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98],"performance","recipe performance","recipe","mm","sentry","harness","app","deterministic","recipes","flow","cold","critical","analysis","tasks","react","browser","layout","screen","network","metamask",{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":104,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":28},"01M70XT2F8A5CQZ81JBP98KSW3","mexc-spot-websocket","MEXC Spot WebSocket workflows for public streams, protobuf handling, depth maintenance, listen keys, and private streams without exposing API keys.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FSKILL.md",[],[108,109,110,111,112,113,114,115,116,117,22,118,119,120],"mexc spot websocket","spot","websocket","api","workflows","public","listen","streams","depth","keys","api keys","maintenance","protobuf",{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":125,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":28},"01M70XT0MBXN3WZY1BKN0CPNKR","mexc-spot-rest","MEXC Spot REST workflows for public data, signed account requests, orders, wallet\u002Fadmin actions, and secure signing without exposing API keys.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FSKILL.md",[],[129,109,130,131,132,133,134,118,22,117,135,136,137,113,138,112,111],"mexc spot rest","REST","admin","signing","Wallet","requests","actions","secure","data","orders",{"skillNo":140,"name":141,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":143,"sourceHtmlUrl":144,"categories":145,"keywords":146,"publishedAt":28},"01M70XSYSVZ2N0N91XCAR2CAK0","mexc-futures-websocket","MEXC Futures WebSocket workflows for public streams, private account\u002Forder\u002Fposition streams, ping\u002Fpong, subscriptions, and depth maintenance.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FSKILL.md",[],[147,148,110,149,150,119,22,151,116,115,113,112],"mexc futures websocket","futures","position","ping","subscriptions",{"skillNo":153,"name":154,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":155,"sourceHtmlUrl":156,"categories":157,"keywords":158,"publishedAt":28},"01M6H0ZWHVE5W04WG2DKQE2SQC","0xklen-version-and-refresh-sanctions-lists","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FSKILL.md",[],[159,160,161],"refresh","0xklen version and refresh sanctions lists","version",{"skillNo":163,"name":164,"nameZh":6,"description":165,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":166,"sourceHtmlUrl":167,"categories":168,"keywords":169,"publishedAt":28},"01M6H0Z28Q3R8J0S799GJQWXM3","verify-tests-cover-changed-lines","Use when a PR adds code that the existing suite may not exercise. Measures patch coverage — the share of newly changed lines hit by tests — before approving.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FSKILL.md",[],[170,171,172,173,174,175,176,24,177,178,179],"tests","cover","verify tests cover changed lines","verify","coverage","pr","exercise","patch","code","share"]