[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6HR5JSX593GFVWW637Z1F2Y":3,"skill-same:01M6HR5JSX593GFVWW637Z1F2Y":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M6HR5JSX593GFVWW637Z1F2Y","spark-optimization",null,"Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.","welflau\u002Fai-dev-system","welflau","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F7010385?v=4","01M6HPTDGF18RFY07M229P8CTM","01M6HPTDMRMZCJPYBFEA9XVVYT",3,"MIT","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"spark","Optimization","spark optimization","data processing pipelines","when","with","processing","scaling","caching","optimize","jobs","pipelines","use","memory","data processing","tuning","data","apache","debugging","apache spark","performance","2026-11-10 10:12:30",false,"---\nname: spark-optimization\ndescription: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.\n---\n\n# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n","# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n",{"name":5,"description":7},"main","18d0e63d1313ca87098f614aa7c1e2dcf6de79c7","backend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization",1,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system","https:\u002F\u002Fgithub.com\u002Fwelflau","2026-11-10 11:14:25",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add welflau\u002Fai-dev-system --skill spark-optimization",{"tool":57,"command":58},"bunx","bunx skills add welflau\u002Fai-dev-system --skill spark-optimization",{"tool":60,"command":61},"pnpm","pnpm dlx skills add welflau\u002Fai-dev-system --skill spark-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002FSKILL.md",true,[],[66,100,130,155,181,205],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":70,"licenseFileUrl":71,"sourceHtmlUrl":72,"categories":73,"keywords":79,"publishedAt":39},"01M6HSTDP4M3JFXEBK91VFB4PA","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Fweb-artifacts-builder\u002FSKILL.md",[74],{"categoryNo":75,"categoryName":76,"taxonomyType":77,"taxonomyTypeDesc":78},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",[80,81,82,83,84,85,86,87,88,89,90,91,92,93,30,94,95,96,97,98,99],"web artifacts builder","web","artifacts","artifacts builder","builder","frontend","css","components","react","routing","state management","state","file","Tailwind","single-file","simple","ui components","multi","for","single",{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":70,"licenseFileUrl":104,"sourceHtmlUrl":105,"categories":106,"keywords":116,"publishedAt":39},"01M6HSDXH0JSP8R3QTNY9XZ8HN","frontend-design","Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML\u002FCSS layouts, or when styling\u002Fbeautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Ffrontend-design\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Ffrontend-design\u002FSKILL.md",[107,108,111],{"categoryNo":75,"categoryName":76,"taxonomyType":77,"taxonomyTypeDesc":78},{"categoryNo":109,"categoryName":110,"taxonomyType":77,"taxonomyTypeDesc":78},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":112,"categoryName":113,"taxonomyType":114,"taxonomyTypeDesc":115},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[85,117,118,86,81,87,88,119,120,121,82,122,123,124,30,125,126,127,128,23,129,22],"design","frontend design","react components","create","code","examples","applications","landing","ui design","pages","aesthetics","web ui","generic",{"skillNo":131,"name":132,"nameZh":6,"description":133,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":70,"licenseFileUrl":134,"sourceHtmlUrl":135,"categories":136,"keywords":138,"publishedAt":39},"01M6HSA77QYVP7A72HREGTDN2C","algorithmic-art","Creating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems. Create original algorithmic art rather than copying existing artists' work to avoid copyright violations.","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Falgorithmic-art\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Falgorithmic-art\u002FSKILL.md",[137],{"categoryNo":109,"categoryName":110,"taxonomyType":77,"taxonomyTypeDesc":78},[139,140,141,120,121,142,143,144,145,30,146,147,148,149,23,150,151,152,153,154,22],"algorithmic art","Art","algorithmic","flow","js","work","interactive","generative art","exploration","systems","users","Existing","request","generative","copyright","fields",{"skillNo":156,"name":157,"nameZh":6,"description":158,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":70,"licenseFileUrl":159,"sourceHtmlUrl":160,"categories":161,"keywords":163,"publishedAt":39},"01M6HS9HE0X390Y6RQKTY1YNGJ","theme-factory","Toolkit for styling artifacts with a theme. These artifacts can be slides, docs, reportings, HTML landing pages, etc. There are 10 pre-set themes with colors\u002Ffonts that you can apply to any artifact that has been creating, or can generate a new theme on-the-fly.","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Ftheme-factory\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Ftheme-factory\u002FSKILL.md",[162],{"categoryNo":109,"categoryName":110,"taxonomyType":77,"taxonomyTypeDesc":78},[164,165,166,167,168,169,82,170,171,172,124,173,174,126,175,23,176,177,178,98,179,180],"theme factory","factory","theme","docs","toolkit","generate","SET","fly","slides","styling","artifact","themes","you","fonts","can","apply","new",{"skillNo":182,"name":183,"nameZh":6,"description":184,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":190,"publishedAt":39},"01M6HS74GASM6FJ77F83EY679E","browsing","Use when you need direct browser control - teaches Chrome DevTools Protocol for controlling existing browser sessions, multi-tab management, form automation, and content extraction via use_browser MCP tool","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fsuperpowers-chrome\u002Fskills\u002Fbrowsing\u002FSKILL.md",[187],{"categoryNo":188,"categoryName":189,"taxonomyType":77,"taxonomyTypeDesc":78},"01JBSEEDCAT000000000000023","Skill与智能体开发",[183,191,192,193,194,195,196,197,198,199,200,30,201,202,203,204,97,176,98,150],"MCP","automation","chrome","sessions","protocol","browser","devtools","chrome devtools","tool","direct","browser control","control","via","form",{"skillNo":206,"name":207,"nameZh":6,"description":208,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":70,"licenseFileUrl":209,"sourceHtmlUrl":210,"categories":211,"keywords":218,"publishedAt":39},"01M6HS6BZT2CKDNQ6YF20DXN4T","slack-gif-creator","Knowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like \"make me a GIF of X doing Y for Slack.\"","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fslack-gif-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fslack-gif-creator\u002FSKILL.md",[212,215],{"categoryNo":213,"categoryName":214,"taxonomyType":77,"taxonomyTypeDesc":78},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":216,"categoryName":217,"taxonomyType":77,"taxonomyTypeDesc":78},"01JBSEEDCAT000000000000016","图像与多媒体",[219,220,221,222,223,224,225,226,227,30,228,149,229,230,231,98,151,22],"slack gif creator","slack","gif","creator","validation","Tools","constraints","Animation","knowledge","me","make","Utilities","concepts"]