[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GYBCDF2A75KBX4W938PDTQ":3,"skill-same:01M6GYBCDF2A75KBX4W938PDTQ":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":35,"linkOnly":36,"markdownContent":37,"markdownBody":38,"frontMatter":39,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":43,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":46,"ownerDescription":47,"updatedAt":48,"install":49,"resources":62},"01M6GYBCDF2A75KBX4W938PDTQ","detect-eval-set-contamination",null,"Use when an eval or benchmark score looks too good, or a test set predates a model's training cutoff. Measures prompt and answer overlap with the training corpus.","jiayaoqijia\u002Fcryptoskill","jiayaoqijia","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1453879?v=4","01M6GMYTX7Q7S504VC6E756ENR","YQ","01M6GMYV1QKJ1CC75MTTGT9PR0",81,"MIT","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-detect-eval-set-contamination\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-detect-eval-set-contamination\u002FSKILL.md",[],[20,21,22,23,24,25,26,27,28,29,30,31,32,33,34],"SET","eval","contamination","detect","detect eval set contamination","test","training","benchmark","model","prompt","use","good","with","answer","when","2026-11-10 10:12:30",false,"---\nname: detect-eval-set-contamination\ndescription: Use when an eval or benchmark score looks too good, or a test set predates a model's training cutoff. Measures prompt and answer overlap with the training corpus.\n---\n\n# Detect Eval-Set Contamination\n\nA model that has seen the test items scores on memory, not ability. Contamination is the default state for any public eval older than the training cutoff, so measure it rather than assume it away.\n\n## Procedure\n\n1. Pin the eval's release date and the model's training cutoff. Any public split published before the cutoff is presumed contaminated until shown otherwise.\n2. Measure n-gram containment: for each item, compute the longest shared word run with the nearest corpus document. A 13-gram match is the standard memorisation signal.\n   ```python\n   def has_match(item, docs, n=13):\n       words = item.split()\n       grams = set(tuple(words[i:i+n]) for i in range(len(words)-n+1))\n       return any(grams & set(tuple(d.split()[i:i+n]) for i in range(len(d.split())-n+1)) for d in docs)\n   ```\n3. Do it on answers, not just questions. Multiple-choice items contaminate through the option text and label order, not only the stem.\n4. Check paraphrase leakage: a model may not recall the exact string but a close variant. Use MinHash near-duplicate detection with a Jaccard threshold of 0.8.\n5. Run the canary test: prepend a random GUID to each item, ask the model to complete the prefix, and see whether it reproduces the gold answer.\n6. Compare contaminated vs clean subsets when you can segment them (e.g. by publication date). The gap is the contamination effect size.\n7. Report contamination as a number — percent of items with a >=13-gram match — not a yes\u002Fno.\n\n```bash\npython3 tools\u002Fcontaminate.py --eval mmlu.jsonl --corpus pile\u002F*.txt --ngram 13 > contam.tsv\n```\n\n## Pitfalls\n\n- Checking only the question stem misses answer-side and option-order leakage.\n- A web-crawled corpus changes under you; record the corpus snapshot hash next to the score.\n- Substring search is not enough — tokenisation differences (whitespace, unicode) hide exact matches.\n- A held-out set you authored after the model's release is clean only if it was never published or used in any fine-tune.\n- Assuming a \"private\" eval is clean misses a downstream fine-tune; check the training data card.\n\n## Verification\n\n    wc -l contam.tsv   # count flagged items; divide by total items for the contamination rate\n\nReport: \"12% of the eval set has a >=13-gram match with the pretraining corpus; the model's 94% drops to 79% on the uncontaminated subset — treat the headline as inflated.\"\n","# Detect Eval-Set Contamination\n\nA model that has seen the test items scores on memory, not ability. Contamination is the default state for any public eval older than the training cutoff, so measure it rather than assume it away.\n\n## Procedure\n\n1. Pin the eval's release date and the model's training cutoff. Any public split published before the cutoff is presumed contaminated until shown otherwise.\n2. Measure n-gram containment: for each item, compute the longest shared word run with the nearest corpus document. A 13-gram match is the standard memorisation signal.\n   ```python\n   def has_match(item, docs, n=13):\n       words = item.split()\n       grams = set(tuple(words[i:i+n]) for i in range(len(words)-n+1))\n       return any(grams & set(tuple(d.split()[i:i+n]) for i in range(len(d.split())-n+1)) for d in docs)\n   ```\n3. Do it on answers, not just questions. Multiple-choice items contaminate through the option text and label order, not only the stem.\n4. Check paraphrase leakage: a model may not recall the exact string but a close variant. Use MinHash near-duplicate detection with a Jaccard threshold of 0.8.\n5. Run the canary test: prepend a random GUID to each item, ask the model to complete the prefix, and see whether it reproduces the gold answer.\n6. Compare contaminated vs clean subsets when you can segment them (e.g. by publication date). The gap is the contamination effect size.\n7. Report contamination as a number — percent of items with a >=13-gram match — not a yes\u002Fno.\n\n```bash\npython3 tools\u002Fcontaminate.py --eval mmlu.jsonl --corpus pile\u002F*.txt --ngram 13 > contam.tsv\n```\n\n## Pitfalls\n\n- Checking only the question stem misses answer-side and option-order leakage.\n- A web-crawled corpus changes under you; record the corpus snapshot hash next to the score.\n- Substring search is not enough — tokenisation differences (whitespace, unicode) hide exact matches.\n- A held-out set you authored after the model's release is clean only if it was never published or used in any fine-tune.\n- Assuming a \"private\" eval is clean misses a downstream fine-tune; check the training data card.\n\n## Verification\n\n    wc -l contam.tsv   # count flagged items; divide by total items for the contamination rate\n\nReport: \"12% of the eval set has a >=13-gram match with the pretraining corpus; the model's 94% drops to 79% on the uncontaminated subset — treat the headline as inflated.\"\n",{"name":5,"description":7},"main","3df1d0d67b30ff15236561f8c0f9f06d33f42dfd","skills\u002Fpayments\u002F0xklen-detect-eval-set-contamination",17,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia","https:\u002F\u002Feth2030.com","Hardcore infra. Building @alt-research and applied cryptography.","2026-11-12 02:28:42",{"source":8,"skillName":5,"commands":50,"prompt":60,"zipAvailable":61},[51,54,57],{"tool":52,"command":53},"npx","npx skills add jiayaoqijia\u002Fcryptoskill --skill detect-eval-set-contamination",{"tool":55,"command":56},"bunx","bunx skills add jiayaoqijia\u002Fcryptoskill --skill detect-eval-set-contamination",{"tool":58,"command":59},"pnpm","pnpm dlx skills add jiayaoqijia\u002Fcryptoskill --skill detect-eval-set-contamination","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-detect-eval-set-contamination\u002FSKILL.md",true,[63,67,73],{"path":64,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":16},"LICENSE","许可证",1063,{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"SOURCE.md","REFERENCE","参考文档",355,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-detect-eval-set-contamination\u002FSOURCE.md",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"TRUST.auto.yaml",2197,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-detect-eval-set-contamination\u002FTRUST.auto.yaml",[78,106,129,147,160,170],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":82,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":35},"01M712C436EQYE43342Q4E7GG1","recipe-performance","Diagnose a specific MetaMask user flow's real-app performance and Sentry critical-user-flow telemetry with deterministic mm-harness recipes. Use for CUF boundaries, cold\u002Fwarm\u002Fbackground-resume behavior, latency regressions, lifecycle tags, duplicate or blocking network requests, long tasks, layout\u002Fpaint cost, screen redraws, React rerenders, or browser trace analysis where tests alone cannot prove the real app.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FSKILL.md",[],[86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105],"performance","recipe performance","recipe","mm","sentry","harness","app","deterministic","recipes","flow","cold","critical","analysis","tasks","react","browser","layout","screen","network","metamask",{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":110,"licenseFileUrl":111,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":35},"01M70XT2F8A5CQZ81JBP98KSW3","mexc-spot-websocket","MEXC Spot WebSocket workflows for public streams, protobuf handling, depth maintenance, listen keys, and private streams without exposing API keys.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FSKILL.md",[],[115,116,117,118,119,120,121,122,123,124,125,126,127,128],"mexc spot websocket","spot","websocket","api","workflows","public","listen","streams","depth","keys","for","api keys","maintenance","protobuf",{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":110,"licenseFileUrl":133,"sourceHtmlUrl":134,"categories":135,"keywords":136,"publishedAt":35},"01M70XT0MBXN3WZY1BKN0CPNKR","mexc-spot-rest","MEXC Spot REST workflows for public data, signed account requests, orders, wallet\u002Fadmin actions, and secure signing without exposing API keys.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FSKILL.md",[],[137,116,138,139,140,141,142,126,125,124,143,144,145,120,146,119,118],"mexc spot rest","REST","admin","signing","Wallet","requests","actions","secure","data","orders",{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":110,"licenseFileUrl":151,"sourceHtmlUrl":152,"categories":153,"keywords":154,"publishedAt":35},"01M70XSYSVZ2N0N91XCAR2CAK0","mexc-futures-websocket","MEXC Futures WebSocket workflows for public streams, private account\u002Forder\u002Fposition streams, ping\u002Fpong, subscriptions, and depth maintenance.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FSKILL.md",[],[155,156,117,157,158,127,125,159,123,122,120,119],"mexc futures websocket","futures","position","ping","subscriptions",{"skillNo":161,"name":162,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":163,"sourceHtmlUrl":164,"categories":165,"keywords":166,"publishedAt":35},"01M6H0ZWHVE5W04WG2DKQE2SQC","0xklen-version-and-refresh-sanctions-lists","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FSKILL.md",[],[167,168,169],"refresh","0xklen version and refresh sanctions lists","version",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":174,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":35},"01M6H0Z28Q3R8J0S799GJQWXM3","verify-tests-cover-changed-lines","Use when a PR adds code that the existing suite may not exercise. Measures patch coverage — the share of newly changed lines hit by tests — before approving.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FSKILL.md",[],[178,179,180,181,182,183,184,30,185,186,187],"tests","cover","verify tests cover changed lines","verify","coverage","pr","exercise","patch","code","share"]