[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7ASKCKC3CR9QDBFQAMQ929J":3,"skill-same:01M7ASKCKC3CR9QDBFQAMQ929J":71},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":6,"updatedAt":34,"install":35,"resources":48},"01M7ASKCKC3CR9QDBFQAMQ929J","gcp-spark",null,"Develops and executes Spark code on Dataproc Clusters and Serverless.\nReads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\nDebugs execution failures.\nUse when:\n- Writing Spark ETL pipelines on GCP.\n- Training or running inference with ML models with spark on GCP.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python scripts that don't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.","saski\u002Farnesto","saski","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F140356?v=4","01M7ASBDSNXTRYDYB4K1PEA1RV","01M7ASBDTAKBDAAV99S1GE7HK6",6,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gcp-spark\ndescription: |\n  Develops and executes Spark code on Dataproc Clusters and Serverless.\n  Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\n  Debugs execution failures.\n  Use when:\n  - Writing Spark ETL pipelines on GCP.\n  - Training or running inference with ML models with spark on GCP.\n  - Managing Spark clusters, jobs, batches, and interactive sessions.\n  Don't use when:\n  - Writing generic Python scripts that don't use Spark.\n  - Performing simple SQL queries that can be done directly in BigQuery.\nlicense: Apache-2.0\nmetadata:\n  version: v2\n  publisher: google\n---\n\n# Spark on Dataproc\n\n> [!IMPORTANT] You MUST ALWAYS follow the Task Execution Workflow when writing\n> spark code.\n\n## Task Execution Workflow\n\n1.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `resources\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names.\n2.  **Generate spark code**:\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `resources\u002Fread_write_data.md` when reading or writing data.\n    *   **ML Tasks**: Refer to `@skill:ml-best-practices` skill and\n        `resources\u002Fml_tasks.md` when generating ML code.\n    *   **Spark Optimizations**: **ALWAYS** refer to\n        `resources\u002Fspark_optimizations.md` when generating spark code and apply\n        optimization whenever applicable.\n3.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `resources\u002Fschema_direct_inspection.md` to verify destination schema.\n4.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first, then\n    compile code using `python3 -m py_compile your-notebook.py`.\n5.  **Execute script**: ONLY when generating a `.py` script refer to\n    `resources\u002Fgcloud_dataproc.md` on writing command to execute generated code\n    on Dataproc. This DOES NOT apply when generating notebooks.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION] Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Avoid toPandas()** Converting a pyspark dataframe to pandas by calling\n    toPandas() can lead to out of memory errors. Only acceptable for building\n    visualizations in Spark 3.5\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Dataproc service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: Query materialization\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS\n*   `roles\u002Fspanner.databaseUser`: Spanner writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `resources\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, and interactive sessions.\n","# Spark on Dataproc\n\n> [!IMPORTANT] You MUST ALWAYS follow the Task Execution Workflow when writing\n> spark code.\n\n## Task Execution Workflow\n\n1.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `resources\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names.\n2.  **Generate spark code**:\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `resources\u002Fread_write_data.md` when reading or writing data.\n    *   **ML Tasks**: Refer to `@skill:ml-best-practices` skill and\n        `resources\u002Fml_tasks.md` when generating ML code.\n    *   **Spark Optimizations**: **ALWAYS** refer to\n        `resources\u002Fspark_optimizations.md` when generating spark code and apply\n        optimization whenever applicable.\n3.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `resources\u002Fschema_direct_inspection.md` to verify destination schema.\n4.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first, then\n    compile code using `python3 -m py_compile your-notebook.py`.\n5.  **Execute script**: ONLY when generating a `.py` script refer to\n    `resources\u002Fgcloud_dataproc.md` on writing command to execute generated code\n    on Dataproc. This DOES NOT apply when generating notebooks.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION] Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Avoid toPandas()** Converting a pyspark dataframe to pandas by calling\n    toPandas() can lead to out of memory errors. Only acceptable for building\n    visualizations in Spark 3.5\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Dataproc service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: Query materialization\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS\n*   `roles\u002Fspanner.databaseUser`: Spanner writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `resources\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, and interactive sessions.\n",{"name":5,"license":14,"metadata":23,"description":26},{"version":24,"publisher":25},"v2","google","Develops and executes Spark code on Dataproc Clusters and Serverless.\nReads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\nDebugs execution failures.\nUse when:\n- Writing Spark ETL pipelines on GCP.\n- Training or running inference with ML models with spark on GCP.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python scripts that don't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.\n","main","3c24fc2c035029b08c7326774bee3fef8fb95566",".agents\u002Fskills\u002Fgcp-spark",0,"https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto","https:\u002F\u002Fgithub.com\u002Fsaski","http:\u002F\u002Fsaski.com","2026-11-12 20:09:16",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add saski\u002Farnesto --skill gcp-spark",{"tool":41,"command":42},"bunx","bunx skills add saski\u002Farnesto --skill gcp-spark",{"tool":44,"command":45},"pnpm","pnpm dlx skills add saski\u002Farnesto --skill gcp-spark","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002FSKILL.md",true,[49,55,59,63,67],{"path":50,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":53,"sourceHtmlUrl":54},"resources\u002Fgcloud_dataproc.md","REFERENCE","参考文档",7819,"https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002Fresources\u002Fgcloud_dataproc.md",{"path":56,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":57,"sourceHtmlUrl":58},"resources\u002Fml_tasks.md",1895,"https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002Fresources\u002Fml_tasks.md",{"path":60,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":61,"sourceHtmlUrl":62},"resources\u002Fread_write_data.md",13570,"https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002Fresources\u002Fread_write_data.md",{"path":64,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":65,"sourceHtmlUrl":66},"resources\u002Fschema_direct_inspection.md",595,"https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002Fresources\u002Fschema_direct_inspection.md",{"path":68,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":69,"sourceHtmlUrl":70},"resources\u002Fspark_optimizations.md",322,"https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgcp-spark\u002Fresources\u002Fspark_optimizations.md",[72,80,87,94,101,108],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":18},"01M7AT7F5FSHNAMFMF01N0QA1X","visualize","Use \u002Fvisualize for creating compact charts, diagrams, or other visuals inline as part of the conversation itself. It chooses among Markdown tables, Mermaid, inline HTML, and other supported in-transcript forms. Use Canvas for durable artifacts outside the transcript that users may revisit, refine, or share.","Unlicense","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.cursor\u002Fskills-cursor\u002Fvisualize\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M7AT79MHSAR3JDDMP7DGERM9","update-cursor-settings","Modify Cursor\u002FVSCode user settings in settings.json. Use when you want to change editor settings, preferences, configuration, themes, font size, tab size, format on save, auto save, keybindings, or any settings.json values.","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.cursor\u002Fskills-cursor\u002Fupdate-cursor-settings\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M7AT73WF3Y22E0FKYAMCCJES","update-cli-config","View and modify Cursor CLI configuration settings in ~\u002F.cursor\u002Fcli-config.json. Use when the user wants to change CLI settings, configure permissions, switch approval mode, enable vim mode, toggle display options, configure sandbox, or manage any CLI preferences.","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.cursor\u002Fskills-cursor\u002Fupdate-cli-config\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M7AT6F3FBP85AJJ5BJNR8B5W","statusline","Configure a custom status line in the CLI. Use when the user mentions status line, statusline, statusLine, CLI status bar, prompt footer customization, or wants to add session context above the prompt.","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.cursor\u002Fskills-cursor\u002Fstatusline\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":18},"01M7AT6D24QFS5XRAFN8S5XJ7W","split-to-prs","Split current work into small reviewable PRs. Use when the user asks to split a chat, set of changes, branch, or PR.","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.cursor\u002Fskills-cursor\u002Fsplit-to-prs\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":18},"01M7AT69FYFN84XZKW452JV5DF","shell","Runs the rest of a \u002Fshell request as a literal shell command. Use only when the user explicitly invokes \u002Fshell and wants the following text executed directly in the terminal.","https:\u002F\u002Fgithub.com\u002Fsaski\u002Farnesto\u002Fblob\u002Fmain\u002F.cursor\u002Fskills-cursor\u002Fshell\u002FSKILL.md",[],[]]