[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6H0DV0V8BTYMHC3WAM1QEA9":3,"skill-same:01M6H0DV0V8BTYMHC3WAM1QEA9":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":33,"linkOnly":34,"markdownContent":35,"markdownBody":36,"frontMatter":37,"refName":38,"commitSha":39,"skillDirectory":40,"forksCount":41,"repositoryHtmlUrl":42,"repositoryHomepageUrl":6,"ownerHtmlUrl":43,"ownerBlogUrl":44,"ownerDescription":45,"updatedAt":46,"install":47,"resources":60},"01M6H0DV0V8BTYMHC3WAM1QEA9","size-an-eval-for-a-meaningful-delta",null,"Use when planning how many eval items to run to detect a difference of a given size. Computes the smallest detectable effect from n so a small gap is not called a win.","jiayaoqijia\u002Fcryptoskill","jiayaoqijia","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1453879?v=4","01M6GMYTX7Q7S504VC6E756ENR","YQ","01M6GMYV1QKJ1CC75MTTGT9PR0",81,"MIT","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-size-an-eval-for-a-meaningful-delta\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-size-an-eval-for-a-meaningful-delta\u002FSKILL.md",[],[20,21,22,23,24,25,26,27,28,29,30,31,32],"eval","delta","size","for","size an eval for a meaningful delta","run","planning","how","effect","use","gap","detect","when","2026-11-10 10:12:30",false,"---\nname: size-an-eval-for-a-meaningful-delta\ndescription: Use when planning how many eval items to run to detect a difference of a given size. Computes the smallest detectable effect from n so a small gap is not called a win.\n---\n\n# Size an Eval for a Meaningful Delta\n\nA 3-point win over 100 items is noise. Before you run, decide the gap you care about, then size the set so the gap exceeds the confidence interval — not the other way around.\n\n## Procedure\n\n1. Fix the decision rule first: \"we ship if accuracy beats baseline by >=5 points\". Convert that to an absolute difference `d`.\n2. Compute the 95% half-width for two independent proportions:\n   `h = 1.96 * sqrt(2 * p*(1-p) \u002F n)`.\n   ```python\n   import math\n   def half_width(n, p=0.5):\n       return 1.96 * math.sqrt(2 * p * (1 - p) \u002F n)\n   for n in (100, 400, 1600, 6400):\n       print(n, round(half_width(n), 4))\n   ```\n3. Solve for n: `n = 2 * p*(1-p) * (1.96\u002Fd)**2`. For d=0.05 and p=0.5 that is ~768 items per arm.\n4. Use paired items when possible — the same items scored by both systems cut variance and can roughly halve n.\n5. Add a difficulty margin: hard slices have lower p and wider intervals; size the smallest slice you will report on, not just the pooled total.\n6. Budget for nondeterminism: if each item is run 3 times, the effective n is the run count, not the item count.\n7. Write the minimum detectable effect into the config so a later reader cannot over-read a small result.\n   ```yaml\n   n_items: 1600\n   mde_abs: 0.035      # gaps below this are not actionable\n   ```\n\n## Pitfalls\n\n- Powering for the pooled metric while deciding on a slice — the slice is underpowered even when the total is fine.\n- Using the observed difference to justify a bigger n after the fact; size before the run.\n- Ignoring base rate: at p=0.99 a 1-point move needs far more items than at p=0.5.\n- Calling a non-significant result \"no effect\" when the interval is wide; it is underpowered, not equal.\n- Forgetting multiple comparisons: testing 20 slices inflates false positives unless the primary slice is pre-declared.\n\n## Verification\n\n    python3 -c \"import math; print('n per arm', round(2*.25*(1.96\u002F0.05)**2))\"   # ~768\n\nReport: \"Decision is a 5-point gap; n=768\u002Farm gives a +\u002F-5% half-width. We ran 1600\u002Farm, MDE +\u002F-3.5% — a 2-point lead is below the MDE and reported as a tie.\"\n","# Size an Eval for a Meaningful Delta\n\nA 3-point win over 100 items is noise. Before you run, decide the gap you care about, then size the set so the gap exceeds the confidence interval — not the other way around.\n\n## Procedure\n\n1. Fix the decision rule first: \"we ship if accuracy beats baseline by >=5 points\". Convert that to an absolute difference `d`.\n2. Compute the 95% half-width for two independent proportions:\n   `h = 1.96 * sqrt(2 * p*(1-p) \u002F n)`.\n   ```python\n   import math\n   def half_width(n, p=0.5):\n       return 1.96 * math.sqrt(2 * p * (1 - p) \u002F n)\n   for n in (100, 400, 1600, 6400):\n       print(n, round(half_width(n), 4))\n   ```\n3. Solve for n: `n = 2 * p*(1-p) * (1.96\u002Fd)**2`. For d=0.05 and p=0.5 that is ~768 items per arm.\n4. Use paired items when possible — the same items scored by both systems cut variance and can roughly halve n.\n5. Add a difficulty margin: hard slices have lower p and wider intervals; size the smallest slice you will report on, not just the pooled total.\n6. Budget for nondeterminism: if each item is run 3 times, the effective n is the run count, not the item count.\n7. Write the minimum detectable effect into the config so a later reader cannot over-read a small result.\n   ```yaml\n   n_items: 1600\n   mde_abs: 0.035      # gaps below this are not actionable\n   ```\n\n## Pitfalls\n\n- Powering for the pooled metric while deciding on a slice — the slice is underpowered even when the total is fine.\n- Using the observed difference to justify a bigger n after the fact; size before the run.\n- Ignoring base rate: at p=0.99 a 1-point move needs far more items than at p=0.5.\n- Calling a non-significant result \"no effect\" when the interval is wide; it is underpowered, not equal.\n- Forgetting multiple comparisons: testing 20 slices inflates false positives unless the primary slice is pre-declared.\n\n## Verification\n\n    python3 -c \"import math; print('n per arm', round(2*.25*(1.96\u002F0.05)**2))\"   # ~768\n\nReport: \"Decision is a 5-point gap; n=768\u002Farm gives a +\u002F-5% half-width. We ran 1600\u002Farm, MDE +\u002F-3.5% — a 2-point lead is below the MDE and reported as a tie.\"\n",{"name":5,"description":7},"main","3df1d0d67b30ff15236561f8c0f9f06d33f42dfd","skills\u002Fpayments\u002F0xklen-size-an-eval-for-a-meaningful-delta",17,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia","https:\u002F\u002Feth2030.com","Hardcore infra. Building @alt-research and applied cryptography.","2026-11-15 02:28:10",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add jiayaoqijia\u002Fcryptoskill --skill size-an-eval-for-a-meaningful-delta",{"tool":53,"command":54},"bunx","bunx skills add jiayaoqijia\u002Fcryptoskill --skill size-an-eval-for-a-meaningful-delta",{"tool":56,"command":57},"pnpm","pnpm dlx skills add jiayaoqijia\u002Fcryptoskill --skill size-an-eval-for-a-meaningful-delta","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-size-an-eval-for-a-meaningful-delta\u002FSKILL.md",true,[61,65,71],{"path":62,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":64,"sourceHtmlUrl":16},"LICENSE","许可证",1063,{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"SOURCE.md","REFERENCE","参考文档",361,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-size-an-eval-for-a-meaningful-delta\u002FSOURCE.md",{"path":72,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":73,"sourceHtmlUrl":74},"TRUST.auto.yaml",2217,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-size-an-eval-for-a-meaningful-delta\u002FTRUST.auto.yaml",[76,104,126,144,157,167],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":80,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":33},"01M712C436EQYE43342Q4E7GG1","recipe-performance","Diagnose a specific MetaMask user flow's real-app performance and Sentry critical-user-flow telemetry with deterministic mm-harness recipes. Use for CUF boundaries, cold\u002Fwarm\u002Fbackground-resume behavior, latency regressions, lifecycle tags, duplicate or blocking network requests, long tasks, layout\u002Fpaint cost, screen redraws, React rerenders, or browser trace analysis where tests alone cannot prove the real app.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FSKILL.md",[],[84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103],"performance","recipe performance","recipe","mm","sentry","harness","app","deterministic","recipes","flow","cold","critical","analysis","tasks","react","browser","layout","screen","network","metamask",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":108,"licenseFileUrl":109,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":33},"01M70XT2F8A5CQZ81JBP98KSW3","mexc-spot-websocket","MEXC Spot WebSocket workflows for public streams, protobuf handling, depth maintenance, listen keys, and private streams without exposing API keys.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FSKILL.md",[],[113,114,115,116,117,118,119,120,121,122,23,123,124,125],"mexc spot websocket","spot","websocket","api","workflows","public","listen","streams","depth","keys","api keys","maintenance","protobuf",{"skillNo":127,"name":128,"nameZh":6,"description":129,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":108,"licenseFileUrl":130,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":33},"01M70XT0MBXN3WZY1BKN0CPNKR","mexc-spot-rest","MEXC Spot REST workflows for public data, signed account requests, orders, wallet\u002Fadmin actions, and secure signing without exposing API keys.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FSKILL.md",[],[134,114,135,136,137,138,139,123,23,122,140,141,142,118,143,117,116],"mexc spot rest","REST","admin","signing","Wallet","requests","actions","secure","data","orders",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":108,"licenseFileUrl":148,"sourceHtmlUrl":149,"categories":150,"keywords":151,"publishedAt":33},"01M70XSYSVZ2N0N91XCAR2CAK0","mexc-futures-websocket","MEXC Futures WebSocket workflows for public streams, private account\u002Forder\u002Fposition streams, ping\u002Fpong, subscriptions, and depth maintenance.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FSKILL.md",[],[152,153,115,154,155,124,23,156,121,120,118,117],"mexc futures websocket","futures","position","ping","subscriptions",{"skillNo":158,"name":159,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":160,"sourceHtmlUrl":161,"categories":162,"keywords":163,"publishedAt":33},"01M6H0ZWHVE5W04WG2DKQE2SQC","0xklen-version-and-refresh-sanctions-lists","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FSKILL.md",[],[164,165,166],"refresh","0xklen version and refresh sanctions lists","version",{"skillNo":168,"name":169,"nameZh":6,"description":170,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":171,"sourceHtmlUrl":172,"categories":173,"keywords":174,"publishedAt":33},"01M6H0Z28Q3R8J0S799GJQWXM3","verify-tests-cover-changed-lines","Use when a PR adds code that the existing suite may not exercise. Measures patch coverage — the share of newly changed lines hit by tests — before approving.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FSKILL.md",[],[175,176,177,178,179,180,181,29,182,183,184],"tests","cover","verify tests cover changed lines","verify","coverage","pr","exercise","patch","code","share"]