[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CVCGJ8KDCD3DE86NCG99Q1":3,"skill-same:01M7CVCGJ8KDCD3DE86NCG99Q1":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":28,"commitSha":29,"skillDirectory":30,"forksCount":14,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":34,"updatedAt":35,"install":36,"resources":49},"01M7CVCGJ8KDCD3DE86NCG99Q1","gcp-spark",null,"Develops, optimizes and runs PySpark\u002FSpark code on Managed Spark (Dataproc\nclusters and Serverless) on Google Cloud.\nUse when:\n- Authoring or running Spark\u002FPySpark notebooks (any kernel), incl. data\n  analysis, reports and visualizations.\n- Writing Spark ETL pipelines or preparing workloads for production.\n- Training or running inference with ML models on Spark.\n- Optimizing PySpark or Spark SQL for performance, memory, or OOM risks.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python that doesn't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.\n- Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting).","Henrique1601\u002FSkiilsToIAs","Henrique1601","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F92006691?v=4","01M7CRXVT0GVXVK80JT1GR687S","Henrique Bezerra Dos Santos","01M7CRXVTN5Y8TMWJ1MDSKCQ1T",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gcp-spark\ndescription: |\n  Develops, optimizes and runs PySpark\u002FSpark code on Managed Spark (Dataproc\n  clusters and Serverless) on Google Cloud.\n  Use when:\n  - Authoring or running Spark\u002FPySpark notebooks (any kernel), incl. data\n    analysis, reports and visualizations.\n  - Writing Spark ETL pipelines or preparing workloads for production.\n  - Training or running inference with ML models on Spark.\n  - Optimizing PySpark or Spark SQL for performance, memory, or OOM risks.\n  - Managing Spark clusters, jobs, batches, and interactive sessions.\n  Don't use when:\n  - Writing generic Python that doesn't use Spark.\n  - Performing simple SQL queries that can be done directly in BigQuery.\n  - Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting).\nlicense: Apache-2.0\nmetadata:\n  version: v24\n  publisher: google\n---\n\n# Managed Spark on Google Cloud\n\n> [!IMPORTANT]\n>\n> You MUST follow the Task Execution Workflow when writing spark code.\n\n## Notebook Generation Environment Rules\n\n> [!CAUTION]\n>\n> When the task is to **generate** a Spark notebook (as opposed to executing\n> one), you **MUST NOT** modify your own local execution environment (i.e. do\n> not run these commands with your shell or other tools). Specifically:\n>\n> -   **NEVER create a virtual environment** (`python -m venv`, `virtualenv`,\n>     `conda create`, `uv venv`, `poetry init\u002Finstall`, etc.).\n> -   **NEVER install packages** (`pip install`, `%pip install`, `uv pip\n>     install`, `conda install`, `poetry add`).\n> -   **NEVER run environment discovery loops** such as activating a venv,\n>     `pip list`, or probing for `ipykernel` \u002F `google-cloud-spark-connect` as a\n>     precondition for writing the notebook.\n>\n> Assume the user will select the appropriate\n> kernel and dependencies after the notebook is generated.\n\n## Task Execution Workflow\n\n1.  **Understand user request**:\n\n    -   When asked to generate a new spark notebook you **MUST** clarify with\n        the user which kernel type will be used: \"Local Python\" (Spark Connect)\n        or \"Remote Spark\". If the user selects \"Local Python\", you MUST\n        additionally clarify which dataprocSessionConfig should be used (scan\n        for existing serverless session templates), and **MUST** initialize the\n        session with `ManagedSparkSession` by adding the following cells to the\n        notebook:\n        ```python\n        from google.cloud.dataproc_v1 import Session\n        # requires google-cloud-spark-connect pypi package for execution\n        from google.cloud.managed_spark_connect import ManagedSparkSession\n\n        session_config = Session()\n        session_config.session_template = (\n            \"projects\u002F\u003CPROJECT_ID>\u002Flocations\u002F\u003CREGION>\u002FsessionTemplates\u002F\u003CTEMPLATE_ID>\"\n        )\n        spark = (\n            ManagedSparkSession.builder.projectId(\"\u003CPROJECT_ID>\")\n            .location(\"\u003CREGION>\")\n            .dataprocSessionConfig(session_config)\n            .getOrCreate()\n        )\n        ```\n2.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `references\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names. Cap GCP data asset discovery attempts\n    at **3 retries max**. Unless explicitly specified, assume\n    that the assets are located in the same project. Avoid scanning for assets\n    across other projects as it can take a long time. If an expected dataset or\n    table does not exist, use `@skill:discovering-gcp-data-assets` to discover\n    all similar tables in the namespace or project.\n\n    *MINOR TYPO RULE*: If there is a minor typo (e.g. `employees` vs\n    `employee`), you can fix the error and proceed.\n\n    *STRICT HALT RULE*: If the discovered table names differ from the requested\n    table by more than a minor typo (e.g. completely different words, prefixes,\n    or suffixes), you must IMMEDIATELY report the missing table and a neutral\n    list of all available alternatives in the same namespace to the user without\n    making any recommendations. You MUST ask the user which alternative to use\n    and then STOP EXECUTING your turn. Do NOT write any Spark code or notebooks.\n    Do NOT proceed with code generation, do NOT add fallback logic to code, and\n    do NOT automatically substitute any alternative table (even if its schema\n    seems to match) without explicit user permission.\n3.  **Verify source accessibility**: verify access\u002Fexistence using `gcloud\n    storage ls gs:\u002F\u002F\u003Cpath-to-dataset>`. If accessing or reading a GCS path fails\n    with a storage error e.g., permission errors like `403\n    Forbidden`\u002F`Forbidden`\u002F`PermissionDenied`, or location errors like `404 Not\n    Found`\u002F`NotFound`\u002F`FileNotFoundException` you should report the error\n    immediately. Either (1) ask the user what to do next, or (2) if asked to\n    execute a notebook, save the notebook with the error output and recommend\n    next steps to resolve the issue. Do NOT scan all buckets for alternative\n    fallback datasets when encountering GCS errors.\n4.  **Generate spark code**:\n\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Spark Session Initialization**:\n\n        > [!IMPORTANT] Initializing a Spark session on Google Cloud can take 2-3\n        > minutes. You MUST inform the user about the potential delay.\n\n        > [!CAUTION] **NEVER** create a local Spark session. The following are\n        > **BANNED**: - `SparkSession.builder.master(\"local\")` -\n        > `pyspark.sql.SparkSession.builder.getOrCreate()` in notebooks or local\n        > Spark Connect scripts when used without `ManagedSparkSession` - Any\n        > `try\u002Fexcept` fallbacks that revert to a local `SparkSession`.\n        >\n        > - **Interactive Notebooks (`.ipynb`) & Spark Connect scripts**: You\n        >   **MUST ALWAYS** use `ManagedSparkSession` from\n        >   `google-cloud-spark-connect` to connect to **Managed Spark\n        >   Serverless**. No exceptions.\n        > - **Dataproc Batch & Cluster jobs (`.py` submitted via `gcloud dataproc batches submit pyspark` or `gcloud dataproc jobs submit pyspark`)**:\n        >   Use standard `SparkSession.builder.appName(\"\u003CAPP_NAME>\").getOrCreate()`\n        >   (never `.master(\"local\")`).\n\n        Refer to `references\u002Fgcloud_dataproc.md` for detailed configuration.\n        Minimal initialization for notebooks \u002F Spark Connect:\n\n        ```python\n        from google.cloud.managed_spark_connect import ManagedSparkSession\n\n        spark = ManagedSparkSession.builder.projectId(\u003CPROJECT_ID>)\n            .location(\u003CREGION>)\n            .getOrCreate()\n        ```\n    *   **Production Logging**: In standalone\n        production PySpark batch jobs and scripts (`.py`), you MUST use the\n        standard Python `logging` module instead of `print()` statements for job\n        lifecycle, progress, and record counts. Do **NOT** use the `logging`\n        module in notebooks (`.ipynb`); use `print()`, `.show()`, and standard\n        cell outputs instead. Configure `logging` for `.py` scripts with\n        timestamps:\n\n        ```python\n        import logging\n\n        logging.basicConfig(\n            level=logging.INFO,\n            format=\"%(asctime)s [%(levelname)s] %(message)s\",\n        )\n        ```\n    *   **Security & Dynamic Variables**: You MUST NOT hardcode plaintext\n        credentials, passwords, or tokens (use Secret Manager). You MUST NOT\n        hardcode environment-specific variables; parameterize project IDs,\n        dataset IDs, table names, and bucket paths via `os.environ.get(...)` or\n        arguments.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `references\u002Fread_write_data.md` when reading or writing data.\n    *   **Machine Learning Tasks**: Refer to `@skill:ml-best-practices` skill and `references\u002Fml_tasks.md` when generating Machine Learning code.\n    *   **Spark Optimizations & Broadcast Joins**: **ALWAYS** refer to\n        `references\u002Fspark_optimizations.md`. When joining a large DataFrame with\n        a small lookup or dimension table, you MUST use `broadcast()` (`from\n        pyspark.sql.functions import broadcast`) on the small table. Treat about\n        10 MB (Spark's default `spark.sql.autoBroadcastJoinThreshold`) as a\n        guide to what counts as small, not a hard limit.\n5.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `references\u002Fschema_direct_inspection.md` to verify destination schema.\n6.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first. Then\n    compile the resulting python script using `python3 -m py_compile\n    your-script.py`. The same can be done for pyspark source code.\n7.  **Execute script or notebook**: When requested to run a job, script,\n    session, or execute notebook cells against Managed Spark, refer to\n    `references\u002Fgcloud_dataproc.md` on how to execute code on Dataproc\n    Serverless using Spark Connect or Dataproc jobs.\n8.  **Follow up with the user**: If a brand new notebook was generated, instruct the user to select the appropriate kernel in the dropdown for cell execution.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION]\n>\n> Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, `broadcast`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Driver memory safety (`toPandas()` \u002F `collect()`)** NEVER call\n    `.toPandas()` or `.collect()` on raw or un-aggregated DataFrames. ALWAYS\n    perform transformations, aggregations (`groupBy().agg()`), or data reduction\n    (`limit()`, `sample()`) in Spark before converting small summaries to Pandas\n    for plotting or display.\n-   [ ] **No inline pip install in Spark batch\u002Fscript jobs**: NEVER run pip\n    install or subprocess package installations inside PySpark batch scripts\n    (the `%pip install` setup cell in generated notebooks is exempt). Pass\n    dependencies using --properties=spark.jars.packages=...,\n    --archives=gs:\u002F\u002F...\u002Fenv.tar.gz#environment, --py-files, or a custom\n    --container-image.\n-   [ ] **Optimization & Pre-Submission Refactoring** Verify the job against\n    `references\u002Fspark_optimizations.md` and follow the confirmation protocol in\n    `references\u002Fspark_refactoring_guide.md`.\n-   [ ] **No full file reads of executed notebooks**: You MUST NOT use `read_file` or\n    generic full file reading tools on executed `.ipynb` notebooks; they often\n    contain massive base64-encoded image outputs that cause excessive token\n    consumption. You MUST inspect execution outputs with cell-scoped reading tools\n    (such as `notebook__read_cell`, `jupyter__read_cell`, or specific line\n    slices).\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Managed Spark (Dataproc) service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: BigLake Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: BigQuery query materialization and load jobs\n*   `roles\u002Fbigquery.dataEditor`: BigQuery table reads and writes (direct and indirect)\n*   `roles\u002Fbigquery.readSessionUser`: BigQuery Storage Read API session creation\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS (data, Iceberg warehouse, and `--deps-bucket` \u002F `temporaryGcsBucket`)\n*   `roles\u002Fspanner.databaseUser`: Spanner reads and writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `references\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, interactive sessions, and Spark Connect sessions.\n\n--------------------------------------------------------------------------------\n\n## Code Optimization & Pre-Submission Verification\n\nBefore submitting any Spark job or Dataproc batch, enforce the pre-submission\nverification protocol in `references\u002Fspark_refactoring_guide.md`: inspect the\ncode for the anti-patterns catalogued in `references\u002Fspark_optimizations.md` and\nobtain explicit user confirmation before applying refactoring.\n\n--------------------------------------------------------------------------------\n\n## Troubleshooting & Root Cause Analysis\n\nFor troubleshooting failed Spark jobs or Dataproc batches, inspecting\u002Ftailing\nGCS driver output logs, analyzing Spark event logs, or performing Root Cause\nAnalysis (RCA), use the `@skill:gcp-spark-troubleshooting` skill.\n","# Managed Spark on Google Cloud\n\n> [!IMPORTANT]\n>\n> You MUST follow the Task Execution Workflow when writing spark code.\n\n## Notebook Generation Environment Rules\n\n> [!CAUTION]\n>\n> When the task is to **generate** a Spark notebook (as opposed to executing\n> one), you **MUST NOT** modify your own local execution environment (i.e. do\n> not run these commands with your shell or other tools). Specifically:\n>\n> -   **NEVER create a virtual environment** (`python -m venv`, `virtualenv`,\n>     `conda create`, `uv venv`, `poetry init\u002Finstall`, etc.).\n> -   **NEVER install packages** (`pip install`, `%pip install`, `uv pip\n>     install`, `conda install`, `poetry add`).\n> -   **NEVER run environment discovery loops** such as activating a venv,\n>     `pip list`, or probing for `ipykernel` \u002F `google-cloud-spark-connect` as a\n>     precondition for writing the notebook.\n>\n> Assume the user will select the appropriate\n> kernel and dependencies after the notebook is generated.\n\n## Task Execution Workflow\n\n1.  **Understand user request**:\n\n    -   When asked to generate a new spark notebook you **MUST** clarify with\n        the user which kernel type will be used: \"Local Python\" (Spark Connect)\n        or \"Remote Spark\". If the user selects \"Local Python\", you MUST\n        additionally clarify which dataprocSessionConfig should be used (scan\n        for existing serverless session templates), and **MUST** initialize the\n        session with `ManagedSparkSession` by adding the following cells to the\n        notebook:\n        ```python\n        from google.cloud.dataproc_v1 import Session\n        # requires google-cloud-spark-connect pypi package for execution\n        from google.cloud.managed_spark_connect import ManagedSparkSession\n\n        session_config = Session()\n        session_config.session_template = (\n            \"projects\u002F\u003CPROJECT_ID>\u002Flocations\u002F\u003CREGION>\u002FsessionTemplates\u002F\u003CTEMPLATE_ID>\"\n        )\n        spark = (\n            ManagedSparkSession.builder.projectId(\"\u003CPROJECT_ID>\")\n            .location(\"\u003CREGION>\")\n            .dataprocSessionConfig(session_config)\n            .getOrCreate()\n        )\n        ```\n2.  **Understand schemas**: **ALWAYS** use `@skill:discovering-gcp-data-assets`\n    skill or `references\u002Fschema_direct_inspection.md` to understand input and\n    output schemas. Include the schema in your thought process BEFORE generating\n    any code. Do NOT guess column names. Cap GCP data asset discovery attempts\n    at **3 retries max**. Unless explicitly specified, assume\n    that the assets are located in the same project. Avoid scanning for assets\n    across other projects as it can take a long time. If an expected dataset or\n    table does not exist, use `@skill:discovering-gcp-data-assets` to discover\n    all similar tables in the namespace or project.\n\n    *MINOR TYPO RULE*: If there is a minor typo (e.g. `employees` vs\n    `employee`), you can fix the error and proceed.\n\n    *STRICT HALT RULE*: If the discovered table names differ from the requested\n    table by more than a minor typo (e.g. completely different words, prefixes,\n    or suffixes), you must IMMEDIATELY report the missing table and a neutral\n    list of all available alternatives in the same namespace to the user without\n    making any recommendations. You MUST ask the user which alternative to use\n    and then STOP EXECUTING your turn. Do NOT write any Spark code or notebooks.\n    Do NOT proceed with code generation, do NOT add fallback logic to code, and\n    do NOT automatically substitute any alternative table (even if its schema\n    seems to match) without explicit user permission.\n3.  **Verify source accessibility**: verify access\u002Fexistence using `gcloud\n    storage ls gs:\u002F\u002F\u003Cpath-to-dataset>`. If accessing or reading a GCS path fails\n    with a storage error e.g., permission errors like `403\n    Forbidden`\u002F`Forbidden`\u002F`PermissionDenied`, or location errors like `404 Not\n    Found`\u002F`NotFound`\u002F`FileNotFoundException` you should report the error\n    immediately. Either (1) ask the user what to do next, or (2) if asked to\n    execute a notebook, save the notebook with the error output and recommend\n    next steps to resolve the issue. Do NOT scan all buckets for alternative\n    fallback datasets when encountering GCS errors.\n4.  **Generate spark code**:\n\n    *   **Output Format**: **ALWAYS** generate code in **Python Notebooks\n        (.ipynb)** format. Generate scripts (.py) only if explicitly requested.\n    *   **Spark Session Initialization**:\n\n        > [!IMPORTANT] Initializing a Spark session on Google Cloud can take 2-3\n        > minutes. You MUST inform the user about the potential delay.\n\n        > [!CAUTION] **NEVER** create a local Spark session. The following are\n        > **BANNED**: - `SparkSession.builder.master(\"local\")` -\n        > `pyspark.sql.SparkSession.builder.getOrCreate()` in notebooks or local\n        > Spark Connect scripts when used without `ManagedSparkSession` - Any\n        > `try\u002Fexcept` fallbacks that revert to a local `SparkSession`.\n        >\n        > - **Interactive Notebooks (`.ipynb`) & Spark Connect scripts**: You\n        >   **MUST ALWAYS** use `ManagedSparkSession` from\n        >   `google-cloud-spark-connect` to connect to **Managed Spark\n        >   Serverless**. No exceptions.\n        > - **Dataproc Batch & Cluster jobs (`.py` submitted via `gcloud dataproc batches submit pyspark` or `gcloud dataproc jobs submit pyspark`)**:\n        >   Use standard `SparkSession.builder.appName(\"\u003CAPP_NAME>\").getOrCreate()`\n        >   (never `.master(\"local\")`).\n\n        Refer to `references\u002Fgcloud_dataproc.md` for detailed configuration.\n        Minimal initialization for notebooks \u002F Spark Connect:\n\n        ```python\n        from google.cloud.managed_spark_connect import ManagedSparkSession\n\n        spark = ManagedSparkSession.builder.projectId(\u003CPROJECT_ID>)\n            .location(\u003CREGION>)\n            .getOrCreate()\n        ```\n    *   **Production Logging**: In standalone\n        production PySpark batch jobs and scripts (`.py`), you MUST use the\n        standard Python `logging` module instead of `print()` statements for job\n        lifecycle, progress, and record counts. Do **NOT** use the `logging`\n        module in notebooks (`.ipynb`); use `print()`, `.show()`, and standard\n        cell outputs instead. Configure `logging` for `.py` scripts with\n        timestamps:\n\n        ```python\n        import logging\n\n        logging.basicConfig(\n            level=logging.INFO,\n            format=\"%(asctime)s [%(levelname)s] %(message)s\",\n        )\n        ```\n    *   **Security & Dynamic Variables**: You MUST NOT hardcode plaintext\n        credentials, passwords, or tokens (use Secret Manager). You MUST NOT\n        hardcode environment-specific variables; parameterize project IDs,\n        dataset IDs, table names, and bucket paths via `os.environ.get(...)` or\n        arguments.\n    *   **Read and Write data**: **ALWAYS** Refer to\n        `references\u002Fread_write_data.md` when reading or writing data.\n    *   **Machine Learning Tasks**: Refer to `@skill:ml-best-practices` skill and `references\u002Fml_tasks.md` when generating Machine Learning code.\n    *   **Spark Optimizations & Broadcast Joins**: **ALWAYS** refer to\n        `references\u002Fspark_optimizations.md`. When joining a large DataFrame with\n        a small lookup or dimension table, you MUST use `broadcast()` (`from\n        pyspark.sql.functions import broadcast`) on the small table. Treat about\n        10 MB (Spark's default `spark.sql.autoBroadcastJoinThreshold`) as a\n        guide to what counts as small, not a hard limit.\n5.  **Verify schema before write**: **ALWAYS** verify that the dataframe and\n    destination schema match, use `df.printSchema()` for dataframe schema and\n    refer to `@skill:discovering-gcp-data-assets` skill or\n    `references\u002Fschema_direct_inspection.md` to verify destination schema.\n6.  **Compile code before executing**: For notebooks convert them to python\n    script using `jupyter nbconvert --to script your-notebook.ipynb` first. Then\n    compile the resulting python script using `python3 -m py_compile\n    your-script.py`. The same can be done for pyspark source code.\n7.  **Execute script or notebook**: When requested to run a job, script,\n    session, or execute notebook cells against Managed Spark, refer to\n    `references\u002Fgcloud_dataproc.md` on how to execute code on Dataproc\n    Serverless using Spark Connect or Dataproc jobs.\n8.  **Follow up with the user**: If a brand new notebook was generated, instruct the user to select the appropriate kernel in the dropdown for cell execution.\n\n--------------------------------------------------------------------------------\n\n## Common Mistakes Checklist\n\n> [!CAUTION]\n>\n> Ensure you verify this checklist to avoid mistakes\n\nBefore submitting a job, verify:\n\n-   [ ] **All imports present** (`col`, `when`, `lit`, `broadcast`, etc. from\n    `pyspark.sql.functions`)\n-   [ ] **`vector_to_array` from correct module** use `from pyspark.ml.functions\n    import vector_to_array` (NOT `pyspark.sql.functions`)\n-   [ ] **DataFrame schema matches target Iceberg table** verify with\n    `df.printSchema()` before writing\n-   [ ] **CSV files read with `header` and `inferSchema`** without these, the\n    header row becomes data and all columns are strings\n-   [ ] **Driver memory safety (`toPandas()` \u002F `collect()`)** NEVER call\n    `.toPandas()` or `.collect()` on raw or un-aggregated DataFrames. ALWAYS\n    perform transformations, aggregations (`groupBy().agg()`), or data reduction\n    (`limit()`, `sample()`) in Spark before converting small summaries to Pandas\n    for plotting or display.\n-   [ ] **No inline pip install in Spark batch\u002Fscript jobs**: NEVER run pip\n    install or subprocess package installations inside PySpark batch scripts\n    (the `%pip install` setup cell in generated notebooks is exempt). Pass\n    dependencies using --properties=spark.jars.packages=...,\n    --archives=gs:\u002F\u002F...\u002Fenv.tar.gz#environment, --py-files, or a custom\n    --container-image.\n-   [ ] **Optimization & Pre-Submission Refactoring** Verify the job against\n    `references\u002Fspark_optimizations.md` and follow the confirmation protocol in\n    `references\u002Fspark_refactoring_guide.md`.\n-   [ ] **No full file reads of executed notebooks**: You MUST NOT use `read_file` or\n    generic full file reading tools on executed `.ipynb` notebooks; they often\n    contain massive base64-encoded image outputs that cause excessive token\n    consumption. You MUST inspect execution outputs with cell-scoped reading tools\n    (such as `notebook__read_cell`, `jupyter__read_cell`, or specific line\n    slices).\n\n--------------------------------------------------------------------------------\n\n## IAM Requirements\n\nThe Managed Spark (Dataproc) service account needs:\n\n*   `roles\u002Fdataproc.worker`: Job execution\n*   `roles\u002Fbiglake.admin`: BigLake Iceberg table management\n*   `roles\u002Fbigquery.jobUser`: BigQuery query materialization and load jobs\n*   `roles\u002Fbigquery.dataEditor`: BigQuery table reads and writes (direct and indirect)\n*   `roles\u002Fbigquery.readSessionUser`: BigQuery Storage Read API session creation\n*   `roles\u002Fstorage.objectUser`: Read\u002Fwrite GCS (data, Iceberg warehouse, and `--deps-bucket` \u002F `temporaryGcsBucket`)\n*   `roles\u002Fspanner.databaseUser`: Spanner reads and writes\n\n--------------------------------------------------------------------------------\n\n## Spark resource management\n\nRefer to `references\u002Fgcloud_dataproc.md` for detailed guidelines on managing\nSpark clusters, jobs, batches, interactive sessions, and Spark Connect sessions.\n\n--------------------------------------------------------------------------------\n\n## Code Optimization & Pre-Submission Verification\n\nBefore submitting any Spark job or Dataproc batch, enforce the pre-submission\nverification protocol in `references\u002Fspark_refactoring_guide.md`: inspect the\ncode for the anti-patterns catalogued in `references\u002Fspark_optimizations.md` and\nobtain explicit user confirmation before applying refactoring.\n\n--------------------------------------------------------------------------------\n\n## Troubleshooting & Root Cause Analysis\n\nFor troubleshooting failed Spark jobs or Dataproc batches, inspecting\u002Ftailing\nGCS driver output logs, analyzing Spark event logs, or performing Root Cause\nAnalysis (RCA), use the `@skill:gcp-spark-troubleshooting` skill.\n",{"name":5,"license":15,"metadata":24,"description":27},{"version":25,"publisher":26},"v24","google","Develops, optimizes and runs PySpark\u002FSpark code on Managed Spark (Dataproc\nclusters and Serverless) on Google Cloud.\nUse when:\n- Authoring or running Spark\u002FPySpark notebooks (any kernel), incl. data\n  analysis, reports and visualizations.\n- Writing Spark ETL pipelines or preparing workloads for production.\n- Training or running inference with ML models on Spark.\n- Optimizing PySpark or Spark SQL for performance, memory, or OOM risks.\n- Managing Spark clusters, jobs, batches, and interactive sessions.\nDon't use when:\n- Writing generic Python that doesn't use Spark.\n- Performing simple SQL queries that can be done directly in BigQuery.\n- Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting).\n","main","a73152352c0612252131e5cf84e91312a6cf17dc","devops-cloud\u002Fgcp-spark","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs","https:\u002F\u002Fgithub.com\u002FHenrique1601","https:\u002F\u002Fbezerraportifolio.netlify.app\u002F","Full Stack Developer | React, Next.js, TypeScript, Node.js & IA | Construindo produtos digitais e soluções SaaS","2026-11-15 02:56:04",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add Henrique1601\u002FSkiilsToIAs --skill gcp-spark",{"tool":42,"command":43},"bunx","bunx skills add Henrique1601\u002FSkiilsToIAs --skill gcp-spark",{"tool":45,"command":46},"pnpm","pnpm dlx skills add Henrique1601\u002FSkiilsToIAs --skill gcp-spark","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002FSKILL.md",true,[50,56,60,64,68,72],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002Fgcloud_dataproc.md","REFERENCE","参考文档",12878,"https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002Freferences\u002Fgcloud_dataproc.md",{"path":57,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fml_tasks.md",3098,"https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002Freferences\u002Fml_tasks.md",{"path":61,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fread_write_data.md",20751,"https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002Freferences\u002Fread_write_data.md",{"path":65,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fschema_direct_inspection.md",1284,"https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002Freferences\u002Fschema_direct_inspection.md",{"path":69,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fspark_optimizations.md",7390,"https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002Freferences\u002Fspark_optimizations.md",{"path":73,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fspark_refactoring_guide.md",2607,"https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fdevops-cloud\u002Fgcp-spark\u002Freferences\u002Fspark_refactoring_guide.md",[77,85,92,99,106,113],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M7E2ANBW9V6AVX5JXNCEAWF8","rag-architect-agent","Agente Arquiteto de RAG, Embeddings & Busca Vetorial. Especialista em design e implementação de pipelines de Retrieval-Augmented Generation de alta fidelidade: chunking semântico, modelagem pgvector no PostgreSQL (Neon\u002FSupabase), busca híbrida (cosseno + BM25), reranking, validação via TypeSafe AI e filosofia Ponytail (anti-overengineering).","MIT","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fcustom-agents\u002Frag-architect-agent\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":19},"01M7E29KKF8R02ZRGYE2MJM67G","rag-architecture-patterns","Padrões arquiteturais e implementação de pipelines RAG (Retrieval-Augmented Generation) de alta fidelidade para qualquer projeto ou sistema. Cobre chunking semântico, embeddings, busca híbrida (pgvector + BM25), reranking, validação via TypeSafe AI e filosofia Ponytail (anti-overengineering).","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Fai-agents\u002Frag-architecture-patterns\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M7CWRA8E3BJ7CPY8N8EG28J1","skill-repair","Use this to fix and re-install agent skills that have failed installation.\nThis skill provides the necessary context and permissions to surgically update\nthe `manifest.json` after a fix has been applied.","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Futilities\u002Fskill-repair\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":19},"01M7CWR66FZP8FA47WVZ4Z2EXC","security-scan","Scan your Claude Code configuration (.claude\u002F directory) for security vulnerabilities, misconfigurations, and injection risks using AgentShield. Checks CLAUDE.md, settings.json, MCP servers, hooks, and agent definitions.","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Futilities\u002Fsecurity-scan\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":19},"01M7CWR0QPY58QPTR1P759TFNG","security-review","Use this skill when adding authentication, handling user input, working with secrets, creating API endpoints, or implementing payment\u002Fsensitive features. Provides comprehensive security checklist and patterns.","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Futilities\u002Fsecurity-review\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":19},"01M7CWQXCE40XWT4FZ3VWZFGDP","security-bounty-hunter","Hunt for exploitable, bounty-worthy security issues in repositories. Focuses on remotely reachable vulnerabilities that qualify for real reports instead of noisy local-only findings.","https:\u002F\u002Fgithub.com\u002FHenrique1601\u002FSkiilsToIAs\u002Fblob\u002Fmain\u002Futilities\u002Fsecurity-bounty-hunter\u002FSKILL.md",[],[]]