[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M75GDZZ4VQ84A4SJ9E5NPFD4":3,"skill-same:01M75GDZZ4VQ84A4SJ9E5NPFD4":50},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":23,"commitSha":24,"skillDirectory":25,"forksCount":26,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":29,"install":30,"resources":43},"01M75GDZZ4VQ84A4SJ9E5NPFD4","spark-optimization",null,"Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.","RudyCity\u002Fsuperagent","RudyCity","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F82188919?v=4","01M75FP5X8AW8ACBH0SH34WS7M","01M75FP5YAHJK8M1Q52BWXB82K",70,"MIT","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fspark-optimization\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: spark-optimization\ndescription: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.\n---\n\n# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor            | Impact                | Solution                      |\n| ----------------- | --------------------- | ----------------------------- |\n| **Shuffle**       | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |\n| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |\n| **Memory**        | GC pressure, spills   | Tune executor memory          |\n| **Partitions**    | Parallelism           | Right-size partitions         |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n\n## Best Practices\n\n### Do's\n\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n","# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor            | Impact                | Solution                      |\n| ----------------- | --------------------- | ----------------------------- |\n| **Shuffle**       | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |\n| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |\n| **Memory**        | GC pressure, spills   | Tune executor memory          |\n| **Partitions**    | Parallelism           | Right-size partitions         |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n\n## Best Practices\n\n### Do's\n\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n",{"name":5,"description":7},"main","62c7af5ced2c21355d148c243edb6d5921cca1a6",".agents\u002Fskills\u002Fspark-optimization",8,"https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent","https:\u002F\u002Fgithub.com\u002FRudyCity","2026-11-14 22:58:49",{"source":8,"skillName":5,"commands":31,"prompt":41,"zipAvailable":42},[32,35,38],{"tool":33,"command":34},"npx","npx skills add RudyCity\u002Fsuperagent --skill spark-optimization",{"tool":36,"command":37},"bunx","bunx skills add RudyCity\u002Fsuperagent --skill spark-optimization",{"tool":39,"command":40},"pnpm","pnpm dlx skills add RudyCity\u002Fsuperagent --skill spark-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fspark-optimization\u002FSKILL.md",true,[44],{"path":45,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":48,"sourceHtmlUrl":49},"references\u002Fdetails.md","REFERENCE","参考文档",9844,"https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fspark-optimization\u002Freferences\u002Fdetails.md",[51,58,65,72,79,86],{"skillNo":52,"name":53,"nameZh":6,"description":54,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":55,"categories":56,"keywords":57,"publishedAt":18},"01M75GP28CM6QCSXSHKFF3NNKS","telegram_bot","Use Telegram Bot when the user asks for Telegram Bot or this provider's API.","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002Fmuse-bridge-kit\u002Fskills\u002Ftelegram-bot\u002FSKILL.md",[],[],{"skillNo":59,"name":60,"nameZh":6,"description":61,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":62,"categories":63,"keywords":64,"publishedAt":18},"01M75GNZGZYM7NWDV6MBP3HS14","superagent-bridge","Operate the Muse\u003C->superagent Telegram bridge: poll the private group for task_request\u002Ftask_result envelopes, validate senders, and reply with task_batch\u002Ftask_done as Muse (the remote brain).","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002Fmuse-bridge-kit\u002Fskills\u002Fsuperagent-bridge\u002FSKILL.md",[],[],{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":18},"01M75GNHN19HJWZ7NAPM5E5JCV","Writing Skills","TDD for process documentation - test with subagents before writing, iterate until bulletproof","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwriting-skills\u002FSKILL.md",[],[],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":18},"01M75GNCQEKVRA4FH161NZPZNB","writing-shape","Take a markdown file of raw material and shape it into an article through a conversational session — drafting candidate openings, growing the piece paragraph by paragraph, arguing about format (lists, tables, callouts, quotes) at each step. Use when the user has a pile of notes, fragments, or a rough draft and wants help turning it into something publishable.","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwriting-shape\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":18},"01M75GNAF9NAYCSTBDZDGMK0DZ","Writing Plans","Create detailed implementation plans with bite-sized tasks for engineers with zero codebase context","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":18},"01M75GN72EV0V5NQAW9CSPFRFH","writing-guidelines","Review docs\u002Fprose for Writing Guidelines compliance. Use when asked to \"review my docs\", \"check writing style\", \"audit prose\", \"review docs voice and tone\", or \"check this page against the writing handbook\".","https:\u002F\u002Fgithub.com\u002FRudyCity\u002Fsuperagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwriting-guidelines\u002FSKILL.md",[],[]]