[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6F1MY0EP6GT3BN079CG7JRN":3,"skill-same:01M6F1MY0EP6GT3BN079CG7JRN":73},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M6F1MY0EP6GT3BN079CG7JRN","spark-optimization",null,"Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.","ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fspark-optimization\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"spark","Optimization","spark optimization","data processing pipelines","with","processing","scaling","caching","optimize","jobs","pipelines","use","memory","data processing","tuning","data","apache","debugging","apache spark","performance","2026-11-10 10:12:30",false,"---\nname: spark-optimization\ndescription: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.\n---\n\n# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n","# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n",{"name":5,"description":7},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fspark-optimization",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill spark-optimization",{"tool":59,"command":60},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill spark-optimization",{"tool":62,"command":63},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill spark-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fspark-optimization\u002FSKILL.md",true,[67],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"skill-report.json","REFERENCE","参考文档",11118,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fspark-optimization\u002Fskill-report.json",[74,88,98,107,113,120],{"skillNo":75,"name":76,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":84,"publishedAt":39},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[79],{"categoryNo":80,"categoryName":81,"taxonomyType":82,"taxonomyTypeDesc":83},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[85,86,87],"model","model hierarchy","model hierarchy skill",{"skillNo":89,"name":90,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":94,"publishedAt":39},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[93],{"categoryNo":80,"categoryName":81,"taxonomyType":82,"taxonomyTypeDesc":83},[95,96,97],"MCP","zoom","zoom mcp",{"skillNo":99,"name":100,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":39},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[104,105,106,96],"sdk","zoom apps sdk","apps",{"skillNo":108,"name":109,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":39},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[109],{"skillNo":114,"name":115,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":119,"publishedAt":39},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[118],{"categoryNo":80,"categoryName":81,"taxonomyType":82,"taxonomyTypeDesc":83},[115],{"skillNo":121,"name":122,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":39},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[122]]