spark-optimization
Diagnose and fix a slow or failing Apache Spark job (PySpark, Spark SQL, Spark 3.x and 4.x) from the Spark UI and query plan, covering shuffles, skew, join strategy, spills, small files, partition sizing and executor memory. Use when a Spark or Databricks job is slow, costly or dies with out-of-memory errors. For scheduling the job use airflow-dag-patterns; for SQL models in a dbt project use dbt-transformation-patterns.
- License 许可
- MIT
- 最近更新
- 2026-11-10
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- spark-optimization
- description
- Diagnose and fix a slow or failing Apache Spark job (PySpark, Spark SQL, Spark 3.x and 4.x) from the Spark UI and query plan, covering shuffles, skew, join strategy, spills, small files, partition sizing and executor memory. Use when a Spark or Databricks job is slow, costly or dies with out-of-memory errors. For scheduling the job use airflow-dag-patterns; for SQL models in a dbt project use dbt-transformation-patterns.
- last_verified
- 1793059200000
02 资源文件 1
- 参考文档references/details.md8.6 KB
03 SKILL.md 正文
Spark Optimization
Find the stage that costs the most, name the cause from evidence, change one thing, and measure again. Examples are checked against PySpark 4.2.
Procedure
- Get the facts. Record the Spark version (
spark.version), the platform (open-source, Databricks, EMR, Dataproc), the cluster size, the input size, and the current runtime of the job. Without a baseline runtime, write "needs measurement" rather than predicting a speedup. - Find the slowest stage in the Spark UI (or the History Server for a finished job). For that stage, read the task-duration percentiles, shuffle read and write, spill (memory and disk), GC time and input size.
- Read the plan. Run
df.explain(mode="formatted")and find eachExchange(a shuffle), the join type (BroadcastHashJoin,SortMergeJoin,ShuffledHashJoin), and whether the scan showsPartitionFiltersandPushedFilters. - Name the cause from the table below and apply its fix. Change one thing per run.
- Re-run on the same input and compare runtime and the stage metrics from step 2. Keep the change only if it helped.
- Stop when the job meets its target, or the slowest stage is limited by input I/O with no spill and no skew. Report what is left.
Report back with the baseline and new runtime, each change with the metric that justified it, and the remaining bottleneck.
Symptom to fix
| Evidence | Cause | Fix |
|---|---|---|
| Max task time far above the median in one stage | Skewed key | Let AQE split it (on by default); if one key dominates, salt it or handle it separately |
SortMergeJoin where one side is small after filtering |
Missed broadcast | F.broadcast(small_df), or raise spark.sql.autoBroadcastJoinThreshold |
| Large spill to disk, long GC | Partitions too big for executor memory | Lower spark.sql.adaptive.advisoryPartitionSizeInBytes or raise executor memory |
| Thousands of tasks of a few MB each | Too many small partitions or files | Compact the input; coalesce before writing |
| Scan reads every file despite a filter | Filter not pushed down | Filter on the partition column, without wrapping it in a function or a UDF |
| Python UDF stage is slow | Row-at-a-time Python | Built-in functions first, then Arrow-based @pandas_udf |
| The same expensive DataFrame recomputed by several actions | No reuse | cache() it once, unpersist() when done |
| Driver out of memory | collect()/toPandas() on large data, or a large broadcast |
Write the result out instead; check broadcast sizes |
Defaults you don't need to set
These are already on in Spark 3.2 and later. Setting them again does nothing; note it in a review rather than adding it:
spark.sql.adaptive.enabled=trueandspark.sql.adaptive.coalescePartitions.enabled=truespark.sql.adaptive.skewJoin.enabled=true(a partition is skewed when it is over 5x the median and over 256 MB)spark.sql.adaptive.localShuffleReader.enabled=truespark.sql.files.maxPartitionBytes=128MB,spark.sql.shuffle.partitions=200(AQE coalesces shuffle partitions down from this number)spark.memory.fraction=0.6,spark.memory.storageFraction=0.5,spark.shuffle.compress=true
spark.serializer=KryoSerializer doesn't speed up DataFrame or SQL jobs: they serialize rows
with Tungsten's binary format. Kryo only affects RDD code and serialized RDD caching.
spark.sql.shuffle.partitions=auto is a Databricks setting; open-source Spark rejects it.
Quick Start
# jobs/inspect_join.py
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("inspect-join").getOrCreate()
orders = spark.range(1_000_000).select(
F.col("id").alias("order_id"),
(F.col("id") % 1000).alias("customer_id"),
(F.rand(seed=1) * 100).alias("amount"),
)
customers = spark.range(1000).select(
F.col("id").alias("customer_id"),
F.when(F.col("id") % 2 == 0, "eu").otherwise("us").alias("region"),
)
totals = (
orders.join(F.broadcast(customers), "customer_id")
.groupBy("region")
.agg(F.sum("amount").alias("revenue"))
)
totals.explain(mode="formatted") # expect BroadcastHashJoin and one Exchange for the groupBy
totals.show()
References
references/details.md: reading the plan and the UI, skew and salting, join strategies and hints, partition sizing and file layout, caching, memory settings, Python UDFs, and Delta or Iceberg table maintenance.
04 同仓库更多 Skills
no-description
暂无简介
good-skill
Fixture skill that satisfies every rule.
broken-link
Links to a reference file that does not exist.
Bad_Name
Name violates the kebab-case rule and does not match the directory.
skill-authoring
Write, review, and validate portable agent skills in SKILL.md format. Use when creating a new skill, editing an existing one, or asked whether a skill follows the Agent Skills spec.
weigh-bags
Record scale readings for each bag — empty (tare), outbound, current, final — and reconcile them against the itemised inventory to find unaccounted weight. Use when the user gives a bag weight, says how heavy something is, weighs their luggage, or asks how close to the limit they are. Appends to bags.yaml and runs scripts/reconcile.py.