[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M77NCH5GD3PB9PWERTJ8CE85":3,"skill-same:01M77NCH5GD3PB9PWERTJ8CE85":52},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M77NCH5GD3PB9PWERTJ8CE85","spark-optimization",null,"Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.","Reihantt6\u002Fsns-myagent","Reihantt6","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F219822095?v=4","01M77KK94PV8B7C2D9XWXNS8DJ","Mohamad Reihan tegar prasetya","01M77KK958QFMTHYXGXFV7KZ79",2,"MIT","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fspark-optimization\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: spark-optimization\ndescription: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.\n---\n\n# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor            | Impact                | Solution                      |\n| ----------------- | --------------------- | ----------------------------- |\n| **Shuffle**       | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |\n| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |\n| **Memory**        | GC pressure, spills   | Tune executor memory          |\n| **Partitions**    | Parallelism           | Right-size partitions         |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n\n## Best Practices\n\n### Do's\n\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n","# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor            | Impact                | Solution                      |\n| ----------------- | --------------------- | ----------------------------- |\n| **Shuffle**       | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |\n| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |\n| **Memory**        | GC pressure, spills   | Tune executor memory          |\n| **Partitions**    | Parallelism           | Right-size partitions         |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references\u002Fdetails.md`. Read that file when the navigation tier above is insufficient.\n\n## Best Practices\n\n### Do's\n\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n",{"name":5,"description":7},"main","58843b159d7a4050d6b785a98b908cce505c0b92",".agents\u002Fskills\u002Fspark-optimization",0,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent","https:\u002F\u002Fgithub.com\u002FReihantt6","reihsns.web.id","2026-08-19 20:08:18",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add Reihantt6\u002Fsns-myagent --skill spark-optimization",{"tool":38,"command":39},"bunx","bunx skills add Reihantt6\u002Fsns-myagent --skill spark-optimization",{"tool":41,"command":42},"pnpm","pnpm dlx skills add Reihantt6\u002Fsns-myagent --skill spark-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fspark-optimization\u002FSKILL.md",true,[46],{"path":47,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":50,"sourceHtmlUrl":51},"references\u002Fdetails.md","REFERENCE","参考文档",9844,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fspark-optimization\u002Freferences\u002Fdetails.md",[53,60,67,74,81,88],{"skillNo":54,"name":55,"nameZh":6,"description":56,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":19},"01M77XDCYZ407XYMF2T4P6F7VT","youtube-transcript","Extract transcripts from YouTube videos. Use when the user asks for a transcript, subtitles, or captions of a YouTube video and provides a YouTube URL (youtube.com\u002Fwatch?v=, youtu.be\u002F, or similar). Supports output with or without timestamps.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fyoutube-transcript\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M77XD7PBC98J6MXGJJ2TJ8DX","youtube-thumbnail-design","YouTube thumbnail design with specific dimensions, contrast rules, and mobile preview optimization. Covers safe zones, text placement, face expression psychology, and A\u002FB testing. Use for: YouTube thumbnails, video cover images, click-through optimization. Triggers: youtube thumbnail, thumbnail design, video thumbnail, click through rate, ctr optimization, youtube cover, video cover image, thumbnail maker, thumbnail tips, youtube design, video preview image","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fyoutube-thumbnail-design\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M77XD2541C8HXJ6RHYZP4NGJ","youtube-short-maker","Make YouTube Shorts that hold attention with Pexo — vertical 9:16, fast-paced, hook in the first 3 seconds. Describe your topic and Pexo writes the script, generates the shots, picks the models, and assembles a finished Short with music and captions. Use for YouTube Shorts: \"youtube short\", \"make a youtube short\", \"shorts video\", \"short-form video\". NOT for long-form or landscape video.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fyoutube-short-maker\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M77XCXKVSYK7KTZ3PQBVVGFD","xcode-project-setup","Safely modifies Xcode projects (.pbxproj) to add Swift Packages and link files. Use this skill whenever an iOS project needs dependencies installed (e.g. Firebase, Alamofire).","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fxcode-project-setup\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M77XCR1DV8ZQK2NHWKVY4EWX","x402","Set up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key. Two setups it works out up front — \"just use it\" (set up a wallet so you or Claude Code can run cloud browser tasks paid from the wallet — Claude writes and runs throwaway scripts, nothing touches your codebase) or \"build it in\" (install the SDK and write the key + code into your project). Walks through wallet setup, funding, .env, and a ~$1 test run. Use when the user asks about x402, pay-per-use, USDC payments, or wants Browser Use Cloud without an API key. For the free-tier signup (reverse-CAPTCHA → API key), use `browser-use cloud signup` or the `cloud` skill instead.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fx402\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M77XCMD3R3ZBZQQSW6E0793B","x-twitter-scraper","Build GitHub Copilot workflows with Xquik X API SDKs, REST endpoints, hosted Apify Actor runs, MCP tools, TweetClaw OpenClaw plugin installs, signed webhooks, tweet search, user lookup, follower exports, media actions, and agent automation.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fx-twitter-scraper\u002FSKILL.md",[],[]]