[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GZ8FVCE7RGQP9YR0YXK4FA":3,"skill-same:01M6GZ8FVCE7RGQP9YR0YXK4FA":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":6,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":53,"updatedAt":54,"install":55,"resources":68},"01M6GZ8FVCE7RGQP9YR0YXK4FA","mine-retrieval-evals-from-query-logs",null,"Use when a hand-written retrieval eval never matches real usage. Builds the query set from logged production queries and labels gold chunks by inspection, so the eval reflects what users actually ask.","jiayaoqijia\u002Fcryptoskill","jiayaoqijia","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1453879?v=4","01M6GMYTX7Q7S504VC6E756ENR","YQ","01M6GMYV1QKJ1CC75MTTGT9PR0",81,"MIT","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-mine-retrieval-evals-from-query-logs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-mine-retrieval-evals-from-query-logs\u002FSKILL.md",[],[20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"logs","mine retrieval evals from query logs","evals","query","mine","retrieval","inspection","when","gold","chunks","labels","hand","usage","users","queries","never","use","ask","eval","SET","Production","2026-11-10 10:12:30",false,"---\nname: mine-retrieval-evals-from-query-logs\ndescription: Use when a hand-written retrieval eval never matches real usage. Builds the query set from logged production queries and labels gold chunks by inspection, so the eval reflects what users actually ask.\n---\n\n# Mine Retrieval Evals from Query Logs\n\nEvals written by the engineer test the engineer's questions. Production queries are the real distribution, including the typos, the underspecified ones, and the ones nobody expected to matter.\n\n## Procedure\n\n1. Export logged queries with their retrieved ids and any click or follow-up signal: `SELECT q, ts, retrieved FROM query_log WHERE ts > now() - interval '30 days'`.\n2. Deduplicate textually near-identical queries by lowercasing, stripping punctuation, and clustering with MinHash (`deduplicate-near-identical-chunks` applied to queries).\n3. Stratify the sample: frequent head queries, a long tail sample, and the zero-result queries separately.\n4. Label gold chunks by reading the retrieved set, not by trusting the ranker: mark the chunk that actually answers each query.\n5. Mark queries the corpus genuinely cannot answer as `unanswerable`, and keep them in the set — they test abstention.\n6. Aim for 30-100 labelled queries to start; small sets cannot distinguish a real gain from noise.\n7. Include the ugliest real queries verbatim: the misspellings and shorthand are where retrieval actually breaks.\n8. Freeze the labelled set into `evals\u002Fretrieval.jsonl` and version it in the repo.\n9. Refresh the set periodically as the query mix drifts, but keep a frozen core so scores stay comparable over time.\n10. Strip PII from logged queries before they enter the repo (see privacy rules); hash user ids rather than storing them.\n\n11. Record for each labelled query whether it came from logs or was authored, so eval mix is visible.\n\n## Pitfalls\n\n- Labelling gold by what the current retriever returned, which bakes in its mistakes as the target.\n- Keeping only the queries the system answered well, so the eval never exercises the failures.\n- Sampling from all-time logs and mixing two product generations of queries into one score.\n- Copying production queries into the repo with emails and account numbers still in them.\n- A set so small that a one-query flip changes the headline number by five points.\n- Letting the eval drift with every label edit, so last month's score is not comparable.\n\n- A log exporter that deduplicates by exact string and keeps every typo variant as a separate query.\n- Labelling from the click log when clicks reflect result position, not relevance.\n\n## Verification\n\n    wc -l evals\u002Fretrieval.jsonl; jq -r '.label' evals\u002Fretrieval.jsonl | sort | uniq -c\n    # passes when the set has >=30 labelled queries, a mix of answerable and unanswerable, and no PII\n\nReport to the user: queries mined, labelled, unanswerable count, the strata sampled, and the PII strip applied.\n","# Mine Retrieval Evals from Query Logs\n\nEvals written by the engineer test the engineer's questions. Production queries are the real distribution, including the typos, the underspecified ones, and the ones nobody expected to matter.\n\n## Procedure\n\n1. Export logged queries with their retrieved ids and any click or follow-up signal: `SELECT q, ts, retrieved FROM query_log WHERE ts > now() - interval '30 days'`.\n2. Deduplicate textually near-identical queries by lowercasing, stripping punctuation, and clustering with MinHash (`deduplicate-near-identical-chunks` applied to queries).\n3. Stratify the sample: frequent head queries, a long tail sample, and the zero-result queries separately.\n4. Label gold chunks by reading the retrieved set, not by trusting the ranker: mark the chunk that actually answers each query.\n5. Mark queries the corpus genuinely cannot answer as `unanswerable`, and keep them in the set — they test abstention.\n6. Aim for 30-100 labelled queries to start; small sets cannot distinguish a real gain from noise.\n7. Include the ugliest real queries verbatim: the misspellings and shorthand are where retrieval actually breaks.\n8. Freeze the labelled set into `evals\u002Fretrieval.jsonl` and version it in the repo.\n9. Refresh the set periodically as the query mix drifts, but keep a frozen core so scores stay comparable over time.\n10. Strip PII from logged queries before they enter the repo (see privacy rules); hash user ids rather than storing them.\n\n11. Record for each labelled query whether it came from logs or was authored, so eval mix is visible.\n\n## Pitfalls\n\n- Labelling gold by what the current retriever returned, which bakes in its mistakes as the target.\n- Keeping only the queries the system answered well, so the eval never exercises the failures.\n- Sampling from all-time logs and mixing two product generations of queries into one score.\n- Copying production queries into the repo with emails and account numbers still in them.\n- A set so small that a one-query flip changes the headline number by five points.\n- Letting the eval drift with every label edit, so last month's score is not comparable.\n\n- A log exporter that deduplicates by exact string and keeps every typo variant as a separate query.\n- Labelling from the click log when clicks reflect result position, not relevance.\n\n## Verification\n\n    wc -l evals\u002Fretrieval.jsonl; jq -r '.label' evals\u002Fretrieval.jsonl | sort | uniq -c\n    # passes when the set has >=30 labelled queries, a mix of answerable and unanswerable, and no PII\n\nReport to the user: queries mined, labelled, unanswerable count, the strata sampled, and the PII strip applied.\n",{"name":5,"description":7},"main","3df1d0d67b30ff15236561f8c0f9f06d33f42dfd","skills\u002Fpayments\u002F0xklen-mine-retrieval-evals-from-query-logs",17,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia","https:\u002F\u002Feth2030.com","Hardcore infra. Building @alt-research and applied cryptography.","2026-11-14 02:27:51",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add jiayaoqijia\u002Fcryptoskill --skill mine-retrieval-evals-from-query-logs",{"tool":61,"command":62},"bunx","bunx skills add jiayaoqijia\u002Fcryptoskill --skill mine-retrieval-evals-from-query-logs",{"tool":64,"command":65},"pnpm","pnpm dlx skills add jiayaoqijia\u002Fcryptoskill --skill mine-retrieval-evals-from-query-logs","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-mine-retrieval-evals-from-query-logs\u002FSKILL.md",true,[69,73,79],{"path":70,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":72,"sourceHtmlUrl":16},"LICENSE","许可证",1063,{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"SOURCE.md","REFERENCE","参考文档",362,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-mine-retrieval-evals-from-query-logs\u002FSOURCE.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"TRUST.auto.yaml",2217,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-mine-retrieval-evals-from-query-logs\u002FTRUST.auto.yaml",[84,112,135,153,166,176],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":88,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":41},"01M712C436EQYE43342Q4E7GG1","recipe-performance","Diagnose a specific MetaMask user flow's real-app performance and Sentry critical-user-flow telemetry with deterministic mm-harness recipes. Use for CUF boundaries, cold\u002Fwarm\u002Fbackground-resume behavior, latency regressions, lifecycle tags, duplicate or blocking network requests, long tasks, layout\u002Fpaint cost, screen redraws, React rerenders, or browser trace analysis where tests alone cannot prove the real app.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FSKILL.md",[],[92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111],"performance","recipe performance","recipe","mm","sentry","harness","app","deterministic","recipes","flow","cold","critical","analysis","tasks","react","browser","layout","screen","network","metamask",{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":116,"licenseFileUrl":117,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":41},"01M70XT2F8A5CQZ81JBP98KSW3","mexc-spot-websocket","MEXC Spot WebSocket workflows for public streams, protobuf handling, depth maintenance, listen keys, and private streams without exposing API keys.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FSKILL.md",[],[121,122,123,124,125,126,127,128,129,130,131,132,133,134],"mexc spot websocket","spot","websocket","api","workflows","public","listen","streams","depth","keys","for","api keys","maintenance","protobuf",{"skillNo":136,"name":137,"nameZh":6,"description":138,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":116,"licenseFileUrl":139,"sourceHtmlUrl":140,"categories":141,"keywords":142,"publishedAt":41},"01M70XT0MBXN3WZY1BKN0CPNKR","mexc-spot-rest","MEXC Spot REST workflows for public data, signed account requests, orders, wallet\u002Fadmin actions, and secure signing without exposing API keys.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FSKILL.md",[],[143,122,144,145,146,147,148,132,131,130,149,150,151,126,152,125,124],"mexc spot rest","REST","admin","signing","Wallet","requests","actions","secure","data","orders",{"skillNo":154,"name":155,"nameZh":6,"description":156,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":116,"licenseFileUrl":157,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":41},"01M70XSYSVZ2N0N91XCAR2CAK0","mexc-futures-websocket","MEXC Futures WebSocket workflows for public streams, private account\u002Forder\u002Fposition streams, ping\u002Fpong, subscriptions, and depth maintenance.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FSKILL.md",[],[161,162,123,163,164,133,131,165,129,128,126,125],"mexc futures websocket","futures","position","ping","subscriptions",{"skillNo":167,"name":168,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":169,"sourceHtmlUrl":170,"categories":171,"keywords":172,"publishedAt":41},"01M6H0ZWHVE5W04WG2DKQE2SQC","0xklen-version-and-refresh-sanctions-lists","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FSKILL.md",[],[173,174,175],"refresh","0xklen version and refresh sanctions lists","version",{"skillNo":177,"name":178,"nameZh":6,"description":179,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":180,"sourceHtmlUrl":181,"categories":182,"keywords":183,"publishedAt":41},"01M6H0Z28Q3R8J0S799GJQWXM3","verify-tests-cover-changed-lines","Use when a PR adds code that the existing suite may not exercise. Measures patch coverage — the share of newly changed lines hit by tests — before approving.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FSKILL.md",[],[184,185,186,187,188,189,190,36,191,192,193],"tests","cover","verify tests cover changed lines","verify","coverage","pr","exercise","patch","code","share"]