[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7B0B9SETA072NAST9G2TZGC":3,"skill-same:01M7B0B9SETA072NAST9G2TZGC":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":13,"repositoryHtmlUrl":30,"repositoryHomepageUrl":6,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":32,"install":33,"resources":46},"01M7B0B9SETA072NAST9G2TZGC","gcp-spark",null,"Develops and executes Spark code on Dataproc Clusters and Serverless.\nReads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\nDebugs execution failures.\nUse when:\n- Writing Spark ETL pipelines on GCP.\n- Training or running inference with ML models with spark on GCP.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python scripts that don't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.","Krishp091205\u002Fgemini-config","Krishp091205","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F307182548?v=4","01M7AXCYY4NQ9WQ7DCT3R5Q5S4","01M7AXCYYR7QFCQQ6BFRKD93YH",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gcp-spark\ndescription: |\n  Develops and executes Spark code on Dataproc Clusters and Serverless.\n  Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\n  Debugs execution failures.\n  Use when:\n  - Writing Spark ETL pipelines on GCP.\n  - Training or running inference with ML models with spark on GCP.\n  - Managing Spark clusters, jobs, batches, and interactive sessions.\n  Don't use when:\n  - Writing generic Python scripts that don't use Spark.\n  - Performing simple SQL queries that can be done directly in BigQuery.\nlicense: Apache-2.0\nmetadata:\n  version: v4\n  publisher: google\n---\n\n# Spark on Dataproc\n\n> [!IMPORTANT]\n>\n> You MUST ALWAYS follow the Task Execution Workflow when writing spark code.\n\n## Task Execution Workflow\n\n1.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `references\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names. Unless explicitly specified, assume\n    that the assets are located in the same project. Avoid scanning for assets\n    across other projects as it can take a long time.\n2.  **Verify source accessibility**: verify access\u002Fexistence using `gcloud\n    storage ls gs:\u002F\u002F\u003Cpath-to-dataset>`. If accessing or reading a GCS path fails\n    with a storage error e.g., permission errors like `403\n    Forbidden`\u002F`Forbidden`\u002F`PermissionDenied`, or location errors like `404 Not\n    Found`\u002F`NotFound`\u002F`FileNotFoundException` you should report the error\n    immediately. Either (1) ask the user what to do next, or (2) if asked to\n    execute a notebook, save the notebook with the error output and recommend\n    next steps to resolve the issue. Do NOT scan all buckets for alternative\n    fallback datasets when encountering GCS errors.\n3.  **Generate spark code**:\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `references\u002Fread_write_data.md` when reading or writing data.\n    *   **ML Tasks**: Refer to `@skill:ml-best-practices` skill and\n        `references\u002Fml_tasks.md` when generating ML code.\n    *   **Spark Optimizations**: **ALWAYS** refer to\n        `references\u002Fspark_optimizations.md` when generating spark code and apply\n        optimization whenever applicable.\n4.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `references\u002Fschema_direct_inspection.md` to verify destination schema.\n5.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first. Then\n    compile the resulting python script using `python3 -m py_compile\n    your-script.py`. The same can be done for pyspark source code.\n6.  **Execute script**: When requested to run a job, script, session refer to\n    `references\u002Fgcloud_dataproc.md` on how to execute generated code on Managed\n    Spark. This DOES NOT apply when generating notebooks.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION]\n>\n> Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Avoid toPandas()** Converting a pyspark dataframe to pandas by calling\n    toPandas() can lead to out of memory errors. Only acceptable for building\n    visualizations in Spark 3.5\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Dataproc service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: Query materialization\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS\n*   `roles\u002Fspanner.databaseUser`: Spanner writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `references\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, and interactive sessions.\n","# Spark on Dataproc\n\n> [!IMPORTANT]\n>\n> You MUST ALWAYS follow the Task Execution Workflow when writing spark code.\n\n## Task Execution Workflow\n\n1.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `references\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names. Unless explicitly specified, assume\n    that the assets are located in the same project. Avoid scanning for assets\n    across other projects as it can take a long time.\n2.  **Verify source accessibility**: verify access\u002Fexistence using `gcloud\n    storage ls gs:\u002F\u002F\u003Cpath-to-dataset>`. If accessing or reading a GCS path fails\n    with a storage error e.g., permission errors like `403\n    Forbidden`\u002F`Forbidden`\u002F`PermissionDenied`, or location errors like `404 Not\n    Found`\u002F`NotFound`\u002F`FileNotFoundException` you should report the error\n    immediately. Either (1) ask the user what to do next, or (2) if asked to\n    execute a notebook, save the notebook with the error output and recommend\n    next steps to resolve the issue. Do NOT scan all buckets for alternative\n    fallback datasets when encountering GCS errors.\n3.  **Generate spark code**:\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `references\u002Fread_write_data.md` when reading or writing data.\n    *   **ML Tasks**: Refer to `@skill:ml-best-practices` skill and\n        `references\u002Fml_tasks.md` when generating ML code.\n    *   **Spark Optimizations**: **ALWAYS** refer to\n        `references\u002Fspark_optimizations.md` when generating spark code and apply\n        optimization whenever applicable.\n4.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `references\u002Fschema_direct_inspection.md` to verify destination schema.\n5.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first. Then\n    compile the resulting python script using `python3 -m py_compile\n    your-script.py`. The same can be done for pyspark source code.\n6.  **Execute script**: When requested to run a job, script, session refer to\n    `references\u002Fgcloud_dataproc.md` on how to execute generated code on Managed\n    Spark. This DOES NOT apply when generating notebooks.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION]\n>\n> Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Avoid toPandas()** Converting a pyspark dataframe to pandas by calling\n    toPandas() can lead to out of memory errors. Only acceptable for building\n    visualizations in Spark 3.5\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Dataproc service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: Query materialization\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS\n*   `roles\u002Fspanner.databaseUser`: Spanner writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `references\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, and interactive sessions.\n",{"name":5,"license":14,"metadata":23,"description":26},{"version":24,"publisher":25},"v4","google","Develops and executes Spark code on Dataproc Clusters and Serverless.\nReads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner.\nDebugs execution failures.\nUse when:\n- Writing Spark ETL pipelines on GCP.\n- Training or running inference with ML models with spark on GCP.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python scripts that don't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.\n","main","1f317365436456f9c1039c414a2f6426caabc74c","config\u002Fskills\u002Fgcp-spark","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config","https:\u002F\u002Fgithub.com\u002FKrishp091205","2026-08-23 02:02:24",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add Krishp091205\u002Fgemini-config --skill gcp-spark",{"tool":39,"command":40},"bunx","bunx skills add Krishp091205\u002Fgemini-config --skill gcp-spark",{"tool":42,"command":43},"pnpm","pnpm dlx skills add Krishp091205\u002Fgemini-config --skill gcp-spark","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002FSKILL.md",true,[47,53,57,61,65],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"references\u002Fgcloud_dataproc.md","REFERENCE","参考文档",7578,"https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002Freferences\u002Fgcloud_dataproc.md",{"path":54,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":55,"sourceHtmlUrl":56},"references\u002Fml_tasks.md",1895,"https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002Freferences\u002Fml_tasks.md",{"path":58,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":59,"sourceHtmlUrl":60},"references\u002Fread_write_data.md",14612,"https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002Freferences\u002Fread_write_data.md",{"path":62,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fschema_direct_inspection.md",1166,"https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002Freferences\u002Fschema_direct_inspection.md",{"path":66,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fspark_optimizations.md",322,"https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fgcp-spark\u002Freferences\u002Fspark_optimizations.md",[70,77,84,91,98,105],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":18},"01M7B0GYVAXB2M99PR4Z3N1WW0","skill-repair","Use this to fix and re-install agent skills that have failed installation.\nThis skill provides the necessary context and permissions to surgically update\nthe `manifest.json` after a fix has been applied.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fskill-repair\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":18},"01M7B0GQ2CKFX5P1DTV802XMAB","notebook-guidance","This skill guides the use of Jupyter notebooks for data analysis, exploration, and visualization, particularly with BigQuery. It outlines best practices for notebook execution and validation (supporting both cell-by-cell execution and full notebook generation depending on tool availability), library installation, and structuring notebooks for clarity. It also covers specific rules for data cleaning, plotting, and integrating with BigQuery SQL and machine learning workflows.\nRelevant when any of the following conditions are true:\n  1. The user request involves a data analysis, data exploration, data visualization, or data insights task that requires multiple steps, queries, or visualizations to answer.\n  2. The user explicitly requests a notebook (.ipynb).\n  3. You are creating, editing, or executing cells in a Jupyter notebook.\n  4. You need to query BigQuery from within a notebook. DO NOT use the Python BigQuery client library; instead, you MUST use the `%%bqsql` magics explained in this skill.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fnotebook-guidance\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":18},"01M7B0GJ1K1JE2K593JZT4CAY9","ml-best-practices","CRITICAL RULE: You MUST use this skill whenever the task involves any machine learning tasks or data analysis.\nUse this skill if the user's prompt or requirements mention any of the following:\n  * Clustering\n  * Classification\n  * Regression\n  * Time series forecasting\n  * Statistical testing\n  * Model comparison\n  * ML\n  * Data analysis\n\nSQL\u002FBigQuery ML HANDOFF: If the user requires a SQL solution, use this skill to dictate the ANALYSIS STEPS (e.g., markdown analysis cells, visualization logic), but defer to `bigquery` for all SQL syntax.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fml-best-practices\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":18},"01M7B0GD7FSN72WCSZ16WEDH42","managing-python-dependencies","Ensures proper Python dependency management, avoiding global `pip install` and\nadhering to project-specific tooling.\n\nUse this skill if any of the following are true:\n  1. Attempting to run `pip install {package_name}`.\n  2. Python packages or dependencies need to be added or modified.\n  3. Initiating a new Python project.\n  4. Creating a new notebook, even if just using BigQuery cells.\n  5. Generating Python code that includes `import` statements for third-party libraries.\n  6. Before executing Python scripts via the terminal to ensure the correct virtual environment is active.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fmanaging-python-dependencies\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":18},"01M7B0G6CC6JP6QTQGPSTBMJTX","google-cloud-storage-basics","Stores, retrieves, and manages data as objects in Cloud Storage (Google Cloud Storage, or GCS) buckets. Use when you need to interact with Cloud Storage — create or configure buckets, upload, download, stream, or transfer data, organize objects with folders, generate signed URLs, control access (IAM, ACLs, public access prevention), set storage classes and tiering (Standard, Nearline, Coldline, Archive), manage cost and lifecycle, protect data (versioning, encryption\u002FCMEK, retention and Bucket Lock, object holds, soft delete), host static websites, trigger Pub\u002FSub notifications on object changes, mount buckets as a file system (gcsfuse), or optimize storage performance at any scale. Covers the gcloud storage \u002F gsutil CLI, JSON and XML APIs, client libraries, Terraform, and Cloud Storage MCP servers. Don't use for block storage (Persistent Disk), data warehousing\u002Fanalytics (BigQuery), or databases (Cloud SQL, Spanner, Bigtable, Firestore).","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fgoogle-cloud-storage-basics\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":18},"01M7B0G2DF32KW9A77S99QBZ4Q","gcs-security-assessment","Assesses security posture, evaluates risks, and checks SAIF compliance for Google Cloud Storage buckets or projects. Use when the user requests security scans, vulnerability checks, or SAIF assessments. Don't use when: The user is asking about non-GCS resources (Compute Engine, GKE, etc.), investigating a live production outage, or asking general security questions not tied to a specific project or bucket.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fgcs-security-assessment\u002FSKILL.md",[],[]]