[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GXZP3JZ5AVF7HMTAD6F4NH":3,"skill-same:01M6GXZP3JZ5AVF7HMTAD6F4NH":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":30,"linkOnly":31,"markdownContent":32,"markdownBody":33,"frontMatter":34,"refName":35,"commitSha":36,"skillDirectory":37,"forksCount":38,"repositoryHtmlUrl":39,"repositoryHomepageUrl":6,"ownerHtmlUrl":40,"ownerBlogUrl":41,"ownerDescription":42,"updatedAt":43,"install":44,"resources":57},"01M6GXZP3JZ5AVF7HMTAD6F4NH","compact-small-files-in-a-lake",null,"Use when a data lake or table format accumulates thousands of tiny files from frequent appends. Compacts them into right-sized files without breaking concurrent readers or the partition layout.","jiayaoqijia\u002Fcryptoskill","jiayaoqijia","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1453879?v=4","01M6GMYTX7Q7S504VC6E756ENR","YQ","01M6GMYV1QKJ1CC75MTTGT9PR0",81,"MIT","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-compact-small-files-in-a-lake\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-compact-small-files-in-a-lake\u002FSKILL.md",[],[20,21,22,23,24,25,26,27,28,29],"files","compact","compact small files in a lake","data","format","table","partition","use","layout","when","2026-11-10 10:12:30",false,"---\nname: compact-small-files-in-a-lake\ndescription: Use when a data lake or table format accumulates thousands of tiny files from frequent appends. Compacts them into right-sized files without breaking concurrent readers or the partition layout.\n---\n\n# Compact Small Files in a Lake\n\nFrequent small appends create millions of tiny files; query planning then spends more time listing files than reading data. Compaction rewrites them into right-sized files while preserving the commit protocol and concurrent reads.\n\n## Procedure\n\n1. Diagnose: list file sizes per partition.\n   `hadoop fs -ls -R s3:\u002F\u002Fbucket\u002Ftable\u002F | awk '{print $5}' | sort -n | head` or read the table format's file-size stats.\n2. Set a target file size (typically 128 MB-1 GB for Parquet in a lake) and a max-files-per-partition threshold.\n3. Compact per partition, not the whole table at once:\n   - Iceberg: `spark.sql(\"CALL catalog.system.rewrite_data_files(table => 'db.t', options => map('target-file-size-bytes','134217728'))\")`.\n   - Delta: `OPTIMIZE db.t WHERE dt='2026-10-01' ZORDER BY (user_id)`.\n4. Use the format's commit protocol so concurrent readers see a consistent snapshot: Iceberg rewrites produce new snapshots, Delta `OPTIMIZE` is ACID. Never delete files by hand.\n5. Schedule compaction after the ingestion window and cap files touched per run to bound cost.\n6. Compact within a partition so files do not straddle partitions and break pruning.\n7. Re-measure file count and bytes-scanned after; then let the snapshot-expiry job reclaim the old files.\n8. For streaming ingest, run compaction on a timer (every N commits or minutes) rather than per micro-batch.\n9. Sort or Z-ORDER on the column queried most, so compaction also reduces scan bytes, not just file count.\n10. Alert when files-per-partition crosses the threshold again, so the compaction lag does not quietly return.\n\n## Pitfalls\n\n- Manual `rm` of old Parquet files under a table format corrupts the transaction log and breaks readers.\n- Compacting the whole table nightly rewrites terabytes to save seconds; compact only hot or small-file-heavy partitions.\n- Ignoring the sort means compaction reduces file count but not scan bytes when the filter column is scattered.\n- Compaction during ingestion races the writer; use the format's concurrency control or a quiet window.\n- Forgetting snapshot expiry leaves storage cost unchanged (old and new both present).\n- A target file size below the block\u002Fstripe size wastes headers.\n- Running compaction as one giant job holds up the table's metadata and blocks concurrent commits.\n- Recompacting the same partition every run because the threshold is set below the achievable size wastes forever.\n\n## Verification\n\n```sh\nspark-sql -e \"SELECT count(*), avg(file_size_in_bytes) FROM db.t.files WHERE partition.dt='2026-10-01'\"\n# a point query's bytes-scanned should drop after compaction\n```\n\nFile count falls within the target, average file size is near the target, and a filtered query scans far fewer bytes. Report partitions compacted, before\u002Fafter file counts, and storage reclaimed after snapshot expiry.","# Compact Small Files in a Lake\n\nFrequent small appends create millions of tiny files; query planning then spends more time listing files than reading data. Compaction rewrites them into right-sized files while preserving the commit protocol and concurrent reads.\n\n## Procedure\n\n1. Diagnose: list file sizes per partition.\n   `hadoop fs -ls -R s3:\u002F\u002Fbucket\u002Ftable\u002F | awk '{print $5}' | sort -n | head` or read the table format's file-size stats.\n2. Set a target file size (typically 128 MB-1 GB for Parquet in a lake) and a max-files-per-partition threshold.\n3. Compact per partition, not the whole table at once:\n   - Iceberg: `spark.sql(\"CALL catalog.system.rewrite_data_files(table => 'db.t', options => map('target-file-size-bytes','134217728'))\")`.\n   - Delta: `OPTIMIZE db.t WHERE dt='2026-10-01' ZORDER BY (user_id)`.\n4. Use the format's commit protocol so concurrent readers see a consistent snapshot: Iceberg rewrites produce new snapshots, Delta `OPTIMIZE` is ACID. Never delete files by hand.\n5. Schedule compaction after the ingestion window and cap files touched per run to bound cost.\n6. Compact within a partition so files do not straddle partitions and break pruning.\n7. Re-measure file count and bytes-scanned after; then let the snapshot-expiry job reclaim the old files.\n8. For streaming ingest, run compaction on a timer (every N commits or minutes) rather than per micro-batch.\n9. Sort or Z-ORDER on the column queried most, so compaction also reduces scan bytes, not just file count.\n10. Alert when files-per-partition crosses the threshold again, so the compaction lag does not quietly return.\n\n## Pitfalls\n\n- Manual `rm` of old Parquet files under a table format corrupts the transaction log and breaks readers.\n- Compacting the whole table nightly rewrites terabytes to save seconds; compact only hot or small-file-heavy partitions.\n- Ignoring the sort means compaction reduces file count but not scan bytes when the filter column is scattered.\n- Compaction during ingestion races the writer; use the format's concurrency control or a quiet window.\n- Forgetting snapshot expiry leaves storage cost unchanged (old and new both present).\n- A target file size below the block\u002Fstripe size wastes headers.\n- Running compaction as one giant job holds up the table's metadata and blocks concurrent commits.\n- Recompacting the same partition every run because the threshold is set below the achievable size wastes forever.\n\n## Verification\n\n```sh\nspark-sql -e \"SELECT count(*), avg(file_size_in_bytes) FROM db.t.files WHERE partition.dt='2026-10-01'\"\n# a point query's bytes-scanned should drop after compaction\n```\n\nFile count falls within the target, average file size is near the target, and a filtered query scans far fewer bytes. Report partitions compacted, before\u002Fafter file counts, and storage reclaimed after snapshot expiry.",{"name":5,"description":7},"main","3df1d0d67b30ff15236561f8c0f9f06d33f42dfd","skills\u002Fpayments\u002F0xklen-compact-small-files-in-a-lake",17,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia","https:\u002F\u002Feth2030.com","Hardcore infra. Building @alt-research and applied cryptography.","2026-11-12 02:28:42",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add jiayaoqijia\u002Fcryptoskill --skill compact-small-files-in-a-lake",{"tool":50,"command":51},"bunx","bunx skills add jiayaoqijia\u002Fcryptoskill --skill compact-small-files-in-a-lake",{"tool":53,"command":54},"pnpm","pnpm dlx skills add jiayaoqijia\u002Fcryptoskill --skill compact-small-files-in-a-lake","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-compact-small-files-in-a-lake\u002FSKILL.md",true,[58,62,68],{"path":59,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":61,"sourceHtmlUrl":16},"LICENSE","许可证",1063,{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"SOURCE.md","REFERENCE","参考文档",355,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-compact-small-files-in-a-lake\u002FSOURCE.md",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"TRUST.auto.yaml",2195,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-compact-small-files-in-a-lake\u002FTRUST.auto.yaml",[73,100,123,140,153,163],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":77,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":30},"01M712C436EQYE43342Q4E7GG1","recipe-performance","Diagnose a specific MetaMask user flow's real-app performance and Sentry critical-user-flow telemetry with deterministic mm-harness recipes. Use for CUF boundaries, cold\u002Fwarm\u002Fbackground-resume behavior, latency regressions, lifecycle tags, duplicate or blocking network requests, long tasks, layout\u002Fpaint cost, screen redraws, React rerenders, or browser trace analysis where tests alone cannot prove the real app.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fwallets\u002Fmetamask-official-recipe-performance\u002FSKILL.md",[],[81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,28,97,98,99],"performance","recipe performance","recipe","mm","sentry","harness","app","deterministic","recipes","flow","cold","critical","analysis","tasks","react","browser","screen","network","metamask",{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":104,"licenseFileUrl":105,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":30},"01M70XT2F8A5CQZ81JBP98KSW3","mexc-spot-websocket","MEXC Spot WebSocket workflows for public streams, protobuf handling, depth maintenance, listen keys, and private streams without exposing API keys.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-websocket\u002FSKILL.md",[],[109,110,111,112,113,114,115,116,117,118,119,120,121,122],"mexc spot websocket","spot","websocket","api","workflows","public","listen","streams","depth","keys","for","api keys","maintenance","protobuf",{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":104,"licenseFileUrl":127,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":30},"01M70XT0MBXN3WZY1BKN0CPNKR","mexc-spot-rest","MEXC Spot REST workflows for public data, signed account requests, orders, wallet\u002Fadmin actions, and secure signing without exposing API keys.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-spot-rest\u002FSKILL.md",[],[131,110,132,133,134,135,136,120,119,118,137,138,23,114,139,113,112],"mexc spot rest","REST","admin","signing","Wallet","requests","actions","secure","orders",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":104,"licenseFileUrl":144,"sourceHtmlUrl":145,"categories":146,"keywords":147,"publishedAt":30},"01M70XSYSVZ2N0N91XCAR2CAK0","mexc-futures-websocket","MEXC Futures WebSocket workflows for public streams, private account\u002Forder\u002Fposition streams, ping\u002Fpong, subscriptions, and depth maintenance.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fexchanges\u002Fmncrftfrcnm-mexc-futures-websocket\u002FSKILL.md",[],[148,149,111,150,151,121,119,152,117,116,114,113],"mexc futures websocket","futures","position","ping","subscriptions",{"skillNo":154,"name":155,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":156,"sourceHtmlUrl":157,"categories":158,"keywords":159,"publishedAt":30},"01M6H0ZWHVE5W04WG2DKQE2SQC","0xklen-version-and-refresh-sanctions-lists","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-version-and-refresh-sanctions-lists\u002FSKILL.md",[],[160,161,162],"refresh","0xklen version and refresh sanctions lists","version",{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":167,"sourceHtmlUrl":168,"categories":169,"keywords":170,"publishedAt":30},"01M6H0Z28Q3R8J0S799GJQWXM3","verify-tests-cover-changed-lines","Use when a PR adds code that the existing suite may not exercise. Measures patch coverage — the share of newly changed lines hit by tests — before approving.","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fpayments\u002F0xklen-verify-tests-cover-changed-lines\u002FSKILL.md",[],[171,172,173,174,175,176,177,27,178,179,180],"tests","cover","verify tests cover changed lines","verify","coverage","pr","exercise","patch","code","share"]