spark-architect
Architect-level Spark job design — partitioning and shuffle mechanics, join strategies and skew, file layout and small-file problems, memory and executor sizing, checkpointing for long jobs, and the tuning order that finds the real bottleneck. Use when large transforms run on Spark.
- License 许可
- MIT
- 最近更新
- 2026-11-13
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- spark-architect
- license
- MIT
- metadata
- { "crusor-path": "data-engineering-architect/storage-architect/spark-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "spark.apache.org/docs/latest/sql-performance-tuning.html; spark.apache.org/docs/latest/tuning.html; docs.databricks.com/en/optimizations", "crusor-triggers": "spark, pyspark, shuffle, data skew, executor memory, broadcast join, partition tuning, small files, spark job slow", "crusor-verified-on": "2026-10-30" }
- description
- Architect-level Spark job design — partitioning and shuffle mechanics, join strategies and skew, file layout and small-file problems, memory and executor sizing, checkpointing for long jobs, and the tuning order that finds the real bottleneck. Use when large transforms run on Spark.
- compatibility
- Apache Spark, Databricks, EMR, Dataproc
03 SKILL.md 正文
Spark Architect
Spark performance is shuffle, skew, and file layout — in that order. Cluster size is usually the last thing to change.
When to use
- Transforms over hundreds of GB to petabytes.
- Existing jobs that are slow, expensive, or failing on memory.
- Designing a lakehouse write path.
When NOT to use
- Data fits comfortably in a warehouse or DuckDB — Spark is overhead.
- Simple scheduled SQL on a warehouse that has its own engine.
Decision 1 — Partitioning
- Target 100–200 MB per output file, 128 MB–1 GB per task input.
- Repartition before writing, not after: the file layout is what downstream pays.
- Partition columns must match query filters; too many partitions = small files.
repartitionfor even distribution,coalescefor reducing without a full shuffle — know which one you are paying for.
Decision 2 — Shuffle
- Every
groupBy,join,distinct, andrepartitionis a shuffle: network, disk, and time. - Reduce shuffle by filtering before joins and projecting only needed columns.
spark.sql.shuffle.partitionsdefault is wrong for most data sizes — tune to data volume ÷ ~200 MB.- Adaptive query execution (
AQE) coalesces partitions and handles skew; enable it and verify it is actually engaging.
Decision 3 — Joins
| Size | Strategy |
|---|---|
| small (< ~100 MB) | broadcast join, no shuffle |
| large × large | sort-merge, tuned partitions |
| skewed key | salting, or AQE skew join |
| range/interval | bucketing or range join carefully |
- Broadcasting something too large causes OOM on the driver/executors — check sizes, do not guess.
- Key skew is the classic: one tenant with 80% of rows. Salt the hot key or let AQE split it.
- Join order matters less than filter pushdown; ensure predicates reach the scan.
Decision 4 — Memory and sizing
- Executor memory must hold a partition plus overhead; more executors do not fix a skewed partition.
- Driver memory for collect-heavy operations; avoid
collect()on large results. - Cache only what is reused multiple times; caching everything evicts what matters.
- Dynamic allocation where available; static sizing over-provisions for the peak stage.
Decision 5 — Reliability
- Idempotent writes: partition overwrite or merge with a deterministic key.
- Checkpoint long lineages/streaming jobs to durable storage.
- Retry with task-level retries; a single bad record should not kill the job — quarantine it.
- Job metrics exported: stage times, shuffle read/write, spill, task failures.
Decision 6 — Tuning order
- Read the Spark UI: which stage dominates?
- Fix file layout and pruning before touching memory.
- Fix skew before adding executors.
- Tune shuffle partitions and AQE next.
- Then memory/executors.
- Only then consider a bigger cluster.
Failure modes at scale
- 300k small files; job spends more time listing than computing.
- One hot key; 199 tasks finish in a minute, one runs for an hour.
- Broadcasting a 2 GB table; executor OOM loop.
collect()on 50M rows to "just check".- Cluster doubled to fix a skew problem that needed salting.
Verification receipts
- Spark UI stage breakdown for the heaviest job, with the bottleneck named.
- File-size and partition-count report for the output table.
- Skew analysis: key distribution for the main join.
- Before/after timings for each tuning change, one at a time.
- Cost per run (DBUs/executor-hours) recorded before and after.
04 同仓库更多 Skills
interaction-patterns-architect
Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.
design-system-ops-architect
Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.
design-handoff-architect
Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.
content-microcopy-architect
Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.
ui-ux-implementation-architect
Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.
test-strategy-architect
Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.