[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7DHEN525V2D8ASQWHK053F9":3,"skill-same:01M7DHEN525V2D8ASQWHK053F9":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":28,"commitSha":29,"skillDirectory":5,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":34,"updatedAt":35,"install":36,"resources":49},"01M7DHEN525V2D8ASQWHK053F9","gcp-spark",null,"Develops and executes Spark code on Dataproc Clusters and Serverless.\nReads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\nDebugs execution failures.\nUse when:\n- Writing Spark ETL pipelines on GCP.\n- Training or running inference with ML models with spark on GCP.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python scripts that don't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.","SaiVatsal\u002FSkills-I-Use","SaiVatsal","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F242597752?v=4","01M7D5YM1T4VXW9203M0GFAHZ1","SaiVatsalPolimera","01M7D5YM2D61ST39XVR2V7B7ZJ",3,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gcp-spark\ndescription: |\n  Develops and executes Spark code on Dataproc Clusters and Serverless.\n  Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\n  Debugs execution failures.\n  Use when:\n  - Writing Spark ETL pipelines on GCP.\n  - Training or running inference with ML models with spark on GCP.\n  - Managing Spark clusters, jobs, batches, and interactive sessions.\n  Don't use when:\n  - Writing generic Python scripts that don't use Spark.\n  - Performing simple SQL queries that can be done directly in BigQuery.\nlicense: Apache-2.0\nmetadata:\n  version: v2\n  publisher: google\n---\n\n# Spark on Dataproc\n\n> [!IMPORTANT]\n>\n> You MUST ALWAYS follow the Task Execution Workflow when writing spark code.\n\n## Task Execution Workflow\n\n1.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `references\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names.\n2.  **Generate spark code**:\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `references\u002Fread_write_data.md` when reading or writing data.\n    *   **ML Tasks**: Refer to `@skill:ml-best-practices` skill and\n        `references\u002Fml_tasks.md` when generating ML code.\n    *   **Spark Optimizations**: **ALWAYS** refer to\n        `references\u002Fspark_optimizations.md` when generating spark code and apply\n        optimization whenever applicable.\n3.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `references\u002Fschema_direct_inspection.md` to verify destination schema.\n4.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first, then\n    compile code using `python3 -m py_compile your-notebook.py`.\n5.  **Execute script**: ONLY when generating a `.py` script refer to\n    `references\u002Fgcloud_dataproc.md` on writing command to execute generated code\n    on Dataproc. This DOES NOT apply when generating notebooks.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION]\n>\n> Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Avoid toPandas()** Converting a pyspark dataframe to pandas by calling\n    toPandas() can lead to out of memory errors. Only acceptable for building\n    visualizations in Spark 3.5\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Dataproc service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: Query materialization\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS\n*   `roles\u002Fspanner.databaseUser`: Spanner writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `references\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, and interactive sessions.\n","# Spark on Dataproc\n\n> [!IMPORTANT]\n>\n> You MUST ALWAYS follow the Task Execution Workflow when writing spark code.\n\n## Task Execution Workflow\n\n1.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `references\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names.\n2.  **Generate spark code**:\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `references\u002Fread_write_data.md` when reading or writing data.\n    *   **ML Tasks**: Refer to `@skill:ml-best-practices` skill and\n        `references\u002Fml_tasks.md` when generating ML code.\n    *   **Spark Optimizations**: **ALWAYS** refer to\n        `references\u002Fspark_optimizations.md` when generating spark code and apply\n        optimization whenever applicable.\n3.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `references\u002Fschema_direct_inspection.md` to verify destination schema.\n4.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first, then\n    compile code using `python3 -m py_compile your-notebook.py`.\n5.  **Execute script**: ONLY when generating a `.py` script refer to\n    `references\u002Fgcloud_dataproc.md` on writing command to execute generated code\n    on Dataproc. This DOES NOT apply when generating notebooks.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION]\n>\n> Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Avoid toPandas()** Converting a pyspark dataframe to pandas by calling\n    toPandas() can lead to out of memory errors. Only acceptable for building\n    visualizations in Spark 3.5\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Dataproc service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: Query materialization\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS\n*   `roles\u002Fspanner.databaseUser`: Spanner writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `references\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, and interactive sessions.\n",{"name":5,"license":15,"metadata":24,"description":27},{"version":25,"publisher":26},"v2","google","Develops and executes Spark code on Dataproc Clusters and Serverless.\nReads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\nDebugs execution failures.\nUse when:\n- Writing Spark ETL pipelines on GCP.\n- Training or running inference with ML models with spark on GCP.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python scripts that don't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.\n","main","8a3b3de73262a4c4db64523949360f3c599c2fb1",0,"https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use","https:\u002F\u002Fgithub.com\u002FSaiVatsal","https:\u002F\u002Fsaivatsal.github.io\u002FPortfolio\u002F","The Longer It Takes The Stronger It Becomes ","2026-09-21 22:45:58",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add SaiVatsal\u002FSkills-I-Use --skill gcp-spark",{"tool":42,"command":43},"bunx","bunx skills add SaiVatsal\u002FSkills-I-Use --skill gcp-spark",{"tool":45,"command":46},"pnpm","pnpm dlx skills add SaiVatsal\u002FSkills-I-Use --skill gcp-spark","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002FSKILL.md",true,[50,56,60,64,68],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002Fgcloud_dataproc.md","REFERENCE","参考文档",7819,"https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002Freferences\u002Fgcloud_dataproc.md",{"path":57,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fml_tasks.md",1895,"https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002Freferences\u002Fml_tasks.md",{"path":61,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fread_write_data.md",13575,"https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002Freferences\u002Fread_write_data.md",{"path":65,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fschema_direct_inspection.md",595,"https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002Freferences\u002Fschema_direct_inspection.md",{"path":69,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fspark_optimizations.md",322,"https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fgcp-spark\u002Freferences\u002Fspark_optimizations.md",[73,81,89,98,105,112],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":77,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7DJZP6E6KMBQV4CZSDPZEZ8","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":85,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M7DJZFTS5TS06K2TV3MDTE67","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fweb-artifacts-builder\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M7DJRK4SRQ908F2TJM6EZ7EV","ui-styling","Create beautiful, accessible user interfaces with shadcn\u002Fui components (built on Radix UI + Tailwind), Tailwind CSS utility-first styling, and canvas-based visual designs. Use when building user interfaces, implementing design systems, creating responsive layouts, adding accessible components (dialogs, dropdowns, forms, tables), customizing themes and colors, implementing dark mode, generating visual designs and posters, or establishing consistent styling patterns across applications.","MIT","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fui-ux-pro-max-skill-main\u002Fcli\u002Fassets\u002Fskills\u002Fui-styling\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fui-ux-pro-max-skill-main\u002Fcli\u002Fassets\u002Fskills\u002Fui-styling\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":19},"01M7DJR902059VEQP23PVM2WA0","design","Comprehensive design skill: brand identity, design tokens, UI styling, logo generation (55 styles, Gemini AI), corporate identity program (50 deliverables, CIP mockups), HTML presentations (Chart.js), banner design (22 styles, social\u002Fads\u002Fweb\u002Fprint), icon design (15 styles, SVG, Gemini 3.1 Pro), social photos (HTML→screenshot, multi-platform). Actions: design logo, create CIP, generate mockups, build slides, design banner, generate icon, create social photos, social media images, brand identity, design system. Platforms: Facebook, Twitter, LinkedIn, YouTube, Instagram, Pinterest, TikTok, Threads, Google Ads.","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fui-ux-pro-max-skill-main\u002Fcli\u002Fassets\u002Fskills\u002Fdesign\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":19},"01M7DJR56QF5N6XYTCPCFHFMNP","design-system","Token architecture, component specifications, and slide generation. Three-layer tokens (primitive→semantic→component), CSS variables, spacing\u002Ftypography scales, component specs, strategic slide creation. Use for design tokens, systematic design, brand-compliant presentations.","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fui-ux-pro-max-skill-main\u002Fcli\u002Fassets\u002Fskills\u002Fdesign-system\u002FSKILL.md",[],[],{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":19},"01M7DJQXB6M0SFF7NX8J5N01AH","banner-design","Design banners for social media, ads, website heroes, creative assets, and print. Multiple art direction options with AI-generated visuals. Actions: design, create, generate banner. Platforms: Facebook, Twitter\u002FX, LinkedIn, YouTube, Instagram, Google Display, website hero, print. Styles: minimalist, gradient, bold typography, photo-based, illustrated, geometric, retro, glassmorphism, 3D, neon, duotone, editorial, collage. Uses ui-ux-pro-max, frontend-design, ai-artist, ai-multimodal skills.","https:\u002F\u002Fgithub.com\u002FSaiVatsal\u002FSkills-I-Use\u002Fblob\u002Fmain\u002Fui-ux-pro-max-skill-main\u002Fcli\u002Fassets\u002Fskills\u002Fbanner-design\u002FSKILL.md",[],[]]