[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79STBPRQD83CRMQ5X4HHTZG":3,"skill-same:01M79STBPRQD83CRMQ5X4HHTZG":53},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":32,"commitSha":33,"skillDirectory":34,"forksCount":14,"repositoryHtmlUrl":35,"repositoryHomepageUrl":6,"ownerHtmlUrl":36,"ownerBlogUrl":6,"ownerDescription":37,"updatedAt":38,"install":39,"resources":52},"01M79STBPRQD83CRMQ5X4HHTZG","spark-architect",null,"Architect-level Spark job design — partitioning and shuffle mechanics, join strategies and skew, file layout and small-file problems, memory and executor sizing, checkpointing for long jobs, and the tuning order that finds the real bottleneck. Use when large transforms run on Spark.","SriAkhilSJ\u002FPulseAIRepo","SriAkhilSJ","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F206201543?v=4","01M79RWFD1Q09B89SXTTKKJY6Z","Sri Akhil SJ","01M79RWFE26NPG0BZ1FRNX376H",0,"MIT","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fdata-engineering-architect\u002Fstorage-architect\u002Fspark-architect\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: spark-architect\ndescription: Architect-level Spark job design — partitioning and shuffle mechanics, join strategies and skew, file layout and small-file problems, memory and executor sizing, checkpointing for long jobs, and the tuning order that finds the real bottleneck. Use when large transforms run on Spark.\nlicense: MIT\ncompatibility: Apache Spark, Databricks, EMR, Dataproc\nmetadata:\n  crusor-domain: data-engineering-architect\n  crusor-path: data-engineering-architect\u002Fstorage-architect\u002Fspark-architect\n  crusor-origin: original\n  crusor-sources: \"spark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html; spark.apache.org\u002Fdocs\u002Flatest\u002Ftuning.html; docs.databricks.com\u002Fen\u002Foptimizations\"\n  crusor-triggers: \"spark, pyspark, shuffle, data skew, executor memory, broadcast join, partition tuning, small files, spark job slow\"\n  crusor-verified-on: \"2026-10-30\"\n---\n\n# Spark Architect\n\nSpark performance is **shuffle, skew, and file layout** — in that order. Cluster\nsize is usually the last thing to change.\n\n## When to use\n\n- Transforms over hundreds of GB to petabytes.\n- Existing jobs that are slow, expensive, or failing on memory.\n- Designing a lakehouse write path.\n\n## When NOT to use\n\n- Data fits comfortably in a warehouse or DuckDB — Spark is overhead.\n- Simple scheduled SQL on a warehouse that has its own engine.\n\n## Decision 1 — Partitioning\n\n- Target 100–200 MB per output file, 128 MB–1 GB per task input.\n- Repartition before writing, not after: the file layout is what downstream pays.\n- Partition columns must match query filters; too many partitions = small files.\n- `repartition` for even distribution, `coalesce` for reducing without a full\n  shuffle — know which one you are paying for.\n\n## Decision 2 — Shuffle\n\n- Every `groupBy`, `join`, `distinct`, and `repartition` is a shuffle: network,\n  disk, and time.\n- Reduce shuffle by filtering before joins and projecting only needed columns.\n- `spark.sql.shuffle.partitions` default is wrong for most data sizes — tune to\n  data volume ÷ ~200 MB.\n- Adaptive query execution (`AQE`) coalesces partitions and handles skew; enable it\n  and verify it is actually engaging.\n\n## Decision 3 — Joins\n\n| Size | Strategy |\n|---|---|\n| small (\u003C ~100 MB) | broadcast join, no shuffle |\n| large × large | sort-merge, tuned partitions |\n| skewed key | salting, or AQE skew join |\n| range\u002Finterval | bucketing or range join carefully |\n\n- Broadcasting something too large causes OOM on the driver\u002Fexecutors — check\n  sizes, do not guess.\n- Key skew is the classic: one tenant with 80% of rows. Salt the hot key or let AQE\n  split it.\n- Join order matters less than filter pushdown; ensure predicates reach the scan.\n\n## Decision 4 — Memory and sizing\n\n- Executor memory must hold a partition plus overhead; more executors do not fix a\n  skewed partition.\n- Driver memory for collect-heavy operations; avoid `collect()` on large results.\n- Cache only what is reused multiple times; caching everything evicts what matters.\n- Dynamic allocation where available; static sizing over-provisions for the peak\n  stage.\n\n## Decision 5 — Reliability\n\n- Idempotent writes: partition overwrite or merge with a deterministic key.\n- Checkpoint long lineages\u002Fstreaming jobs to durable storage.\n- Retry with task-level retries; a single bad record should not kill the job —\n  quarantine it.\n- Job metrics exported: stage times, shuffle read\u002Fwrite, spill, task failures.\n\n## Decision 6 — Tuning order\n\n1. Read the Spark UI: which stage dominates?\n2. Fix file layout and pruning before touching memory.\n3. Fix skew before adding executors.\n4. Tune shuffle partitions and AQE next.\n5. Then memory\u002Fexecutors.\n6. Only then consider a bigger cluster.\n\n## Failure modes at scale\n\n- 300k small files; job spends more time listing than computing.\n- One hot key; 199 tasks finish in a minute, one runs for an hour.\n- Broadcasting a 2 GB table; executor OOM loop.\n- `collect()` on 50M rows to \"just check\".\n- Cluster doubled to fix a skew problem that needed salting.\n\n## Verification receipts\n\n1. Spark UI stage breakdown for the heaviest job, with the bottleneck named.\n2. File-size and partition-count report for the output table.\n3. Skew analysis: key distribution for the main join.\n4. Before\u002Fafter timings for each tuning change, one at a time.\n5. Cost per run (DBUs\u002Fexecutor-hours) recorded before and after.\n","# Spark Architect\n\nSpark performance is **shuffle, skew, and file layout** — in that order. Cluster\nsize is usually the last thing to change.\n\n## When to use\n\n- Transforms over hundreds of GB to petabytes.\n- Existing jobs that are slow, expensive, or failing on memory.\n- Designing a lakehouse write path.\n\n## When NOT to use\n\n- Data fits comfortably in a warehouse or DuckDB — Spark is overhead.\n- Simple scheduled SQL on a warehouse that has its own engine.\n\n## Decision 1 — Partitioning\n\n- Target 100–200 MB per output file, 128 MB–1 GB per task input.\n- Repartition before writing, not after: the file layout is what downstream pays.\n- Partition columns must match query filters; too many partitions = small files.\n- `repartition` for even distribution, `coalesce` for reducing without a full\n  shuffle — know which one you are paying for.\n\n## Decision 2 — Shuffle\n\n- Every `groupBy`, `join`, `distinct`, and `repartition` is a shuffle: network,\n  disk, and time.\n- Reduce shuffle by filtering before joins and projecting only needed columns.\n- `spark.sql.shuffle.partitions` default is wrong for most data sizes — tune to\n  data volume ÷ ~200 MB.\n- Adaptive query execution (`AQE`) coalesces partitions and handles skew; enable it\n  and verify it is actually engaging.\n\n## Decision 3 — Joins\n\n| Size | Strategy |\n|---|---|\n| small (\u003C ~100 MB) | broadcast join, no shuffle |\n| large × large | sort-merge, tuned partitions |\n| skewed key | salting, or AQE skew join |\n| range\u002Finterval | bucketing or range join carefully |\n\n- Broadcasting something too large causes OOM on the driver\u002Fexecutors — check\n  sizes, do not guess.\n- Key skew is the classic: one tenant with 80% of rows. Salt the hot key or let AQE\n  split it.\n- Join order matters less than filter pushdown; ensure predicates reach the scan.\n\n## Decision 4 — Memory and sizing\n\n- Executor memory must hold a partition plus overhead; more executors do not fix a\n  skewed partition.\n- Driver memory for collect-heavy operations; avoid `collect()` on large results.\n- Cache only what is reused multiple times; caching everything evicts what matters.\n- Dynamic allocation where available; static sizing over-provisions for the peak\n  stage.\n\n## Decision 5 — Reliability\n\n- Idempotent writes: partition overwrite or merge with a deterministic key.\n- Checkpoint long lineages\u002Fstreaming jobs to durable storage.\n- Retry with task-level retries; a single bad record should not kill the job —\n  quarantine it.\n- Job metrics exported: stage times, shuffle read\u002Fwrite, spill, task failures.\n\n## Decision 6 — Tuning order\n\n1. Read the Spark UI: which stage dominates?\n2. Fix file layout and pruning before touching memory.\n3. Fix skew before adding executors.\n4. Tune shuffle partitions and AQE next.\n5. Then memory\u002Fexecutors.\n6. Only then consider a bigger cluster.\n\n## Failure modes at scale\n\n- 300k small files; job spends more time listing than computing.\n- One hot key; 199 tasks finish in a minute, one runs for an hour.\n- Broadcasting a 2 GB table; executor OOM loop.\n- `collect()` on 50M rows to \"just check\".\n- Cluster doubled to fix a skew problem that needed salting.\n\n## Verification receipts\n\n1. Spark UI stage breakdown for the heaviest job, with the bottleneck named.\n2. File-size and partition-count report for the output table.\n3. Skew analysis: key distribution for the main join.\n4. Before\u002Fafter timings for each tuning change, one at a time.\n5. Cost per run (DBUs\u002Fexecutor-hours) recorded before and after.\n",{"name":5,"license":15,"metadata":24,"description":7,"compatibility":31},{"crusor-path":25,"crusor-domain":26,"crusor-origin":27,"crusor-sources":28,"crusor-triggers":29,"crusor-verified-on":30},"data-engineering-architect\u002Fstorage-architect\u002Fspark-architect","data-engineering-architect","original","spark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html; spark.apache.org\u002Fdocs\u002Flatest\u002Ftuning.html; docs.databricks.com\u002Fen\u002Foptimizations","spark, pyspark, shuffle, data skew, executor memory, broadcast join, partition tuning, small files, spark job slow","2026-10-30","Apache Spark, Databricks, EMR, Dataproc","main","8fa4e240be65b98472c9b2d3a505ade2aea1a5c6","skills\u002Fdata-engineering-architect\u002Fstorage-architect\u002Fspark-architect","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ","Studying in CMR institute of Technology\r\n2024 - 2028","2026-11-13 21:02:23",{"source":8,"skillName":5,"commands":40,"prompt":50,"zipAvailable":51},[41,44,47],{"tool":42,"command":43},"npx","npx skills add SriAkhilSJ\u002FPulseAIRepo --skill spark-architect",{"tool":45,"command":46},"bunx","bunx skills add SriAkhilSJ\u002FPulseAIRepo --skill spark-architect",{"tool":48,"command":49},"pnpm","pnpm dlx skills add SriAkhilSJ\u002FPulseAIRepo --skill spark-architect","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fdata-engineering-architect\u002Fstorage-architect\u002Fspark-architect\u002FSKILL.md",true,[],[54,61,68,75,82,89],{"skillNo":55,"name":56,"nameZh":6,"description":57,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":19},"01M79TTAJNACCNXCEN3649CMMD","interaction-patterns-architect","Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fwebsite-architect\u002Fui-ux-implementation-architect\u002Finteraction-patterns-architect\u002FSKILL.md",[],[],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":19},"01M79TT2ESKB1TKZBYY9KYJHV9","design-system-ops-architect","Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fwebsite-architect\u002Fui-ux-implementation-architect\u002Fdesign-system-ops-architect\u002FSKILL.md",[],[],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":19},"01M79TSV0PWGNK3PRX9P4JDR9N","design-handoff-architect","Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fwebsite-architect\u002Fui-ux-implementation-architect\u002Fdesign-handoff-architect\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M79TSJS9BFXNP696FMJ9GKDA","content-microcopy-architect","Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fwebsite-architect\u002Fui-ux-implementation-architect\u002Fcontent-microcopy-architect\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M79TSCNBST3N1M68V1KPH9HQ","ui-ux-implementation-architect","Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fwebsite-architect\u002Fui-ux-implementation-architect\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M79TS4N6A4D2QQ433VXS5D0R","test-strategy-architect","Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.","https:\u002F\u002Fgithub.com\u002FSriAkhilSJ\u002FPulseAIRepo\u002Fblob\u002Fmain\u002Fskills\u002Fwebsite-architect\u002Ftesting-and-deployment-architect\u002Ftest-strategy-architect\u002FSKILL.md",[],[]]