[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7F0EQ65ABCKT651RXY3CVMG":3,"skill-same:01M7F0EQ65ABCKT651RXY3CVMG":53},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":14,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7F0EQ65ABCKT651RXY3CVMG","spark-optimization",null,"Diagnose and fix a slow or failing Apache Spark job (PySpark, Spark SQL, Spark 3.x and 4.x) from the Spark UI and query plan, covering shuffles, skew, join strategy, spills, small files, partition sizing and executor memory. Use when a Spark or Databricks job is slow, costly or dies with out-of-memory errors. For scheduling the job use airflow-dag-patterns; for SQL models in a dbt project use dbt-transformation-patterns.","shotcowboystyle\u002Fag-skills","shotcowboystyle","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F62471?v=4","01M7EZTCFCZJ6K3MYW8BPM4X6R","Curtis Blanton","01M7EZTCG1S5M04J90A931X9BX",0,"MIT","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002FSKILL.md",[],[],"2026-11-16 10:05:34",false,"---\nname: spark-optimization\ndescription: Diagnose and fix a slow or failing Apache Spark job (PySpark, Spark SQL, Spark 3.x and 4.x) from the Spark UI and query plan, covering shuffles, skew, join strategy, spills, small files, partition sizing and executor memory. Use when a Spark or Databricks job is slow, costly or dies with out-of-memory errors. For scheduling the job use airflow-dag-patterns; for SQL models in a dbt project use dbt-transformation-patterns.\nlast_verified: 2026-10-27\n---\n\n# Spark Optimization\n\nFind the stage that costs the most, name the cause from evidence, change one thing, and\nmeasure again. Examples are checked against PySpark 4.2.\n\n## Procedure\n\n1. **Get the facts.** Record the Spark version (`spark.version`), the platform (open-source,\n   Databricks, EMR, Dataproc), the cluster size, the input size, and the current runtime of\n   the job. Without a baseline runtime, write \"needs measurement\" rather than predicting a\n   speedup.\n2. **Find the slowest stage** in the Spark UI (or the History Server for a finished job). For\n   that stage, read the task-duration percentiles, shuffle read and write, spill (memory and\n   disk), GC time and input size.\n3. **Read the plan.** Run `df.explain(mode=\"formatted\")` and find each `Exchange` (a shuffle),\n   the join type (`BroadcastHashJoin`, `SortMergeJoin`, `ShuffledHashJoin`), and whether the\n   scan shows `PartitionFilters` and `PushedFilters`.\n4. **Name the cause** from the table below and apply its fix. Change one thing per run.\n5. **Re-run on the same input** and compare runtime and the stage metrics from step 2. Keep the\n   change only if it helped.\n6. **Stop when** the job meets its target, or the slowest stage is limited by input I\u002FO with no\n   spill and no skew. Report what is left.\n\nReport back with the baseline and new runtime, each change with the metric that justified it,\nand the remaining bottleneck.\n\n## Symptom to fix\n\n| Evidence | Cause | Fix |\n| --- | --- | --- |\n| Max task time far above the median in one stage | Skewed key | Let AQE split it (on by default); if one key dominates, salt it or handle it separately |\n| `SortMergeJoin` where one side is small after filtering | Missed broadcast | `F.broadcast(small_df)`, or raise `spark.sql.autoBroadcastJoinThreshold` |\n| Large spill to disk, long GC | Partitions too big for executor memory | Lower `spark.sql.adaptive.advisoryPartitionSizeInBytes` or raise executor memory |\n| Thousands of tasks of a few MB each | Too many small partitions or files | Compact the input; `coalesce` before writing |\n| Scan reads every file despite a filter | Filter not pushed down | Filter on the partition column, without wrapping it in a function or a UDF |\n| Python UDF stage is slow | Row-at-a-time Python | Built-in functions first, then Arrow-based `@pandas_udf` |\n| The same expensive DataFrame recomputed by several actions | No reuse | `cache()` it once, `unpersist()` when done |\n| Driver out of memory | `collect()`\u002F`toPandas()` on large data, or a large broadcast | Write the result out instead; check broadcast sizes |\n\n## Defaults you don't need to set\n\nThese are already on in Spark 3.2 and later. Setting them again does nothing; note it in a\nreview rather than adding it:\n\n- `spark.sql.adaptive.enabled=true` and `spark.sql.adaptive.coalescePartitions.enabled=true`\n- `spark.sql.adaptive.skewJoin.enabled=true` (a partition is skewed when it is over 5x the\n  median and over 256 MB)\n- `spark.sql.adaptive.localShuffleReader.enabled=true`\n- `spark.sql.files.maxPartitionBytes=128MB`, `spark.sql.shuffle.partitions=200` (AQE coalesces\n  shuffle partitions down from this number)\n- `spark.memory.fraction=0.6`, `spark.memory.storageFraction=0.5`, `spark.shuffle.compress=true`\n\n`spark.serializer=KryoSerializer` doesn't speed up DataFrame or SQL jobs: they serialize rows\nwith Tungsten's binary format. Kryo only affects RDD code and serialized RDD caching.\n`spark.sql.shuffle.partitions=auto` is a Databricks setting; open-source Spark rejects it.\n\n## Quick Start\n\n```python\n# jobs\u002Finspect_join.py\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\nspark = SparkSession.builder.appName(\"inspect-join\").getOrCreate()\n\norders = spark.range(1_000_000).select(\n    F.col(\"id\").alias(\"order_id\"),\n    (F.col(\"id\") % 1000).alias(\"customer_id\"),\n    (F.rand(seed=1) * 100).alias(\"amount\"),\n)\ncustomers = spark.range(1000).select(\n    F.col(\"id\").alias(\"customer_id\"),\n    F.when(F.col(\"id\") % 2 == 0, \"eu\").otherwise(\"us\").alias(\"region\"),\n)\n\ntotals = (\n    orders.join(F.broadcast(customers), \"customer_id\")\n    .groupBy(\"region\")\n    .agg(F.sum(\"amount\").alias(\"revenue\"))\n)\ntotals.explain(mode=\"formatted\")  # expect BroadcastHashJoin and one Exchange for the groupBy\ntotals.show()\n```\n\n## References\n\n- `references\u002Fdetails.md`: reading the plan and the UI, skew and salting, join strategies and\n  hints, partition sizing and file layout, caching, memory settings, Python UDFs, and Delta or\n  Iceberg table maintenance.\n","# Spark Optimization\n\nFind the stage that costs the most, name the cause from evidence, change one thing, and\nmeasure again. Examples are checked against PySpark 4.2.\n\n## Procedure\n\n1. **Get the facts.** Record the Spark version (`spark.version`), the platform (open-source,\n   Databricks, EMR, Dataproc), the cluster size, the input size, and the current runtime of\n   the job. Without a baseline runtime, write \"needs measurement\" rather than predicting a\n   speedup.\n2. **Find the slowest stage** in the Spark UI (or the History Server for a finished job). For\n   that stage, read the task-duration percentiles, shuffle read and write, spill (memory and\n   disk), GC time and input size.\n3. **Read the plan.** Run `df.explain(mode=\"formatted\")` and find each `Exchange` (a shuffle),\n   the join type (`BroadcastHashJoin`, `SortMergeJoin`, `ShuffledHashJoin`), and whether the\n   scan shows `PartitionFilters` and `PushedFilters`.\n4. **Name the cause** from the table below and apply its fix. Change one thing per run.\n5. **Re-run on the same input** and compare runtime and the stage metrics from step 2. Keep the\n   change only if it helped.\n6. **Stop when** the job meets its target, or the slowest stage is limited by input I\u002FO with no\n   spill and no skew. Report what is left.\n\nReport back with the baseline and new runtime, each change with the metric that justified it,\nand the remaining bottleneck.\n\n## Symptom to fix\n\n| Evidence | Cause | Fix |\n| --- | --- | --- |\n| Max task time far above the median in one stage | Skewed key | Let AQE split it (on by default); if one key dominates, salt it or handle it separately |\n| `SortMergeJoin` where one side is small after filtering | Missed broadcast | `F.broadcast(small_df)`, or raise `spark.sql.autoBroadcastJoinThreshold` |\n| Large spill to disk, long GC | Partitions too big for executor memory | Lower `spark.sql.adaptive.advisoryPartitionSizeInBytes` or raise executor memory |\n| Thousands of tasks of a few MB each | Too many small partitions or files | Compact the input; `coalesce` before writing |\n| Scan reads every file despite a filter | Filter not pushed down | Filter on the partition column, without wrapping it in a function or a UDF |\n| Python UDF stage is slow | Row-at-a-time Python | Built-in functions first, then Arrow-based `@pandas_udf` |\n| The same expensive DataFrame recomputed by several actions | No reuse | `cache()` it once, `unpersist()` when done |\n| Driver out of memory | `collect()`\u002F`toPandas()` on large data, or a large broadcast | Write the result out instead; check broadcast sizes |\n\n## Defaults you don't need to set\n\nThese are already on in Spark 3.2 and later. Setting them again does nothing; note it in a\nreview rather than adding it:\n\n- `spark.sql.adaptive.enabled=true` and `spark.sql.adaptive.coalescePartitions.enabled=true`\n- `spark.sql.adaptive.skewJoin.enabled=true` (a partition is skewed when it is over 5x the\n  median and over 256 MB)\n- `spark.sql.adaptive.localShuffleReader.enabled=true`\n- `spark.sql.files.maxPartitionBytes=128MB`, `spark.sql.shuffle.partitions=200` (AQE coalesces\n  shuffle partitions down from this number)\n- `spark.memory.fraction=0.6`, `spark.memory.storageFraction=0.5`, `spark.shuffle.compress=true`\n\n`spark.serializer=KryoSerializer` doesn't speed up DataFrame or SQL jobs: they serialize rows\nwith Tungsten's binary format. Kryo only affects RDD code and serialized RDD caching.\n`spark.sql.shuffle.partitions=auto` is a Databricks setting; open-source Spark rejects it.\n\n## Quick Start\n\n```python\n# jobs\u002Finspect_join.py\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\nspark = SparkSession.builder.appName(\"inspect-join\").getOrCreate()\n\norders = spark.range(1_000_000).select(\n    F.col(\"id\").alias(\"order_id\"),\n    (F.col(\"id\") % 1000).alias(\"customer_id\"),\n    (F.rand(seed=1) * 100).alias(\"amount\"),\n)\ncustomers = spark.range(1000).select(\n    F.col(\"id\").alias(\"customer_id\"),\n    F.when(F.col(\"id\") % 2 == 0, \"eu\").otherwise(\"us\").alias(\"region\"),\n)\n\ntotals = (\n    orders.join(F.broadcast(customers), \"customer_id\")\n    .groupBy(\"region\")\n    .agg(F.sum(\"amount\").alias(\"revenue\"))\n)\ntotals.explain(mode=\"formatted\")  # expect BroadcastHashJoin and one Exchange for the groupBy\ntotals.show()\n```\n\n## References\n\n- `references\u002Fdetails.md`: reading the plan and the UI, skew and salting, join strategies and\n  hints, partition sizing and file layout, caching, memory settings, Python UDFs, and Delta or\n  Iceberg table maintenance.\n",{"name":5,"description":7,"last_verified":24},1793059200000,"main","f846d9a97082ff30d613b55a085cef425d822473","plugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle","https:\u002F\u002Fshotcowboystyle.github.io","Building smarter with AI, automation & modern tools. Exploring how far one person can go without a big team using today’s AI.","2026-11-10 04:58:27",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add shotcowboystyle\u002Fag-skills --skill spark-optimization",{"tool":39,"command":40},"bunx","bunx skills add shotcowboystyle\u002Fag-skills --skill spark-optimization",{"tool":42,"command":43},"pnpm","pnpm dlx skills add shotcowboystyle\u002Fag-skills --skill spark-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002FSKILL.md",true,[47],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"references\u002Fdetails.md","REFERENCE","参考文档",8761,"https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002Freferences\u002Fdetails.md",[54,60,67,74,81,88],{"skillNo":55,"name":56,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":19},"01M7F1YRY5S5QRJ1V0BJS130QX","no-description","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fno-description\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M7F1Y9BV2FJNQJ3NB32JS5CM","good-skill","Fixture skill that satisfies every rule.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fgood-skill\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M7F1XXBQ3NQRXZSS0KAGAR9V","broken-link","Links to a reference file that does not exist.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fbroken-link\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7F1XBX4R46NR9W0HXQ7XCDX","Bad_Name","Name violates the kebab-case rule and does not match the directory.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fbad-name\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7F1WNAF8WNEQ7S5HKT5VK0D","skill-authoring","Write, review, and validate portable agent skills in SKILL.md format. Use when creating a new skill, editing an existing one, or asked whether a skill follows the Agent Skills spec.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fskills\u002Fskill-authoring\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7F1W30TC0XFNSF7YBRGETK2","weigh-bags","Record scale readings for each bag — empty (tare), outbound, current, final — and reconcile them against the itemised inventory to find unaccounted weight. Use when the user gives a bag weight, says how heavy something is, weighs their luggage, or asks how close to the limit they are. Appends to bags.yaml and runs scripts\u002Freconcile.py.","https:\u002F\u002Fgithub.com\u002Fshotcowboystyle\u002Fag-skills\u002Fblob\u002Fmain\u002Fplugins\u002Ftravel-packing-assistant\u002Fskills\u002Fweigh-bags\u002FSKILL.md",[],[]]