gcp-spark
Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
- License 许可
- Apache-2.0
- 最近更新
- 2026-08-01
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- gcp-spark
- license
- Apache-2.0
- metadata
- { "version": "v2", "publisher": "google" }
- description
- Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
02 资源文件 5
- 参考文档resources/gcloud_dataproc.md7.6 KB
- 参考文档resources/ml_tasks.md1.9 KB
- 参考文档resources/read_write_data.md13.3 KB
- 参考文档resources/schema_direct_inspection.md595 B
- 参考文档resources/spark_optimizations.md322 B
03 SKILL.md 正文
Spark on Dataproc
[!IMPORTANT] You MUST ALWAYS follow the Task Execution Workflow when writing spark code.
Task Execution Workflow
- Understand schemas: ALWAYS use
@skill:discovering-gcp-data-assetsskill orresources/schema_direct_inspection.mdto understand input and output schemas. Include the schema in your thought process BEFORE generating any code. Do NOT guess column names. - Generate spark code:
- Output Format: ALWAYS generate code in Python Notebooks (.ipynb) format. Generate scripts (.py) only if explicitly requested.
- Read and Write data: ALWAYS Refer to
resources/read_write_data.mdwhen reading or writing data. - ML Tasks: Refer to
@skill:ml-best-practicesskill andresources/ml_tasks.mdwhen generating ML code. - Spark Optimizations: ALWAYS refer to
resources/spark_optimizations.mdwhen generating spark code and apply optimization whenever applicable.
- Verify schema before write: ALWAYS verify that the dataframe and
destination schema match, use
df.printSchema()for dataframe schema and refer to@skill:discovering-gcp-data-assetsskill orresources/schema_direct_inspection.mdto verify destination schema. - Compile code before executing: For notebooks convert them to python
script using
jupyter nbconvert --to script your-notebook.ipynbfirst, then compile code usingpython3 -m py_compile your-notebook.py. - Execute script: ONLY when generating a
.pyscript refer toresources/gcloud_dataproc.mdon writing command to execute generated code on Dataproc. This DOES NOT apply when generating notebooks.
Common Mistakes Checklist
[!CAUTION] Ensure you verify this checklist to avoid mistakes
Before submitting a job, verify:
- [ ] All imports present (
col,when,lit, etc. frompyspark.sql.functions) - [ ]
vector_to_arrayfrom correct module usefrom pyspark.ml.functions import vector_to_array(NOTpyspark.sql.functions) - [ ] DataFrame schema matches target Iceberg table verify with
df.printSchema()before writing - [ ] CSV files read with
headerandinferSchemawithout these, the header row becomes data and all columns are strings - [ ] Avoid toPandas() Converting a pyspark dataframe to pandas by calling toPandas() can lead to out of memory errors. Only acceptable for building visualizations in Spark 3.5
IAM Requirements
The Dataproc service account needs:
roles/dataproc.worker: Job executionroles/biglake.admin: Iceberg table managementroles/bigquery.jobUser: Query materializationroles/storage.objectUser: Read/write GCSroles/spanner.databaseUser: Spanner writes
Spark resource management
Refer to resources/gcloud_dataproc.md for detailed guidelines on managing
Spark clusters, jobs, batches, and interactive sessions.
04 同仓库更多 Skills
zustand-store-ts
Create Zustand stores following established patterns with proper TypeScript types and middleware.
zoom-automation
Automate Zoom meeting creation, management, recordings, webinars, and participant tracking via Rube MCP (Composio). Always search tools first for current schemas.
zoho-crm-automation
Automate Zoho CRM tasks via Rube MCP (Composio): create/update records, search contacts, manage leads, and convert leads. Always search tools first for current schemas.
zod-validation-expert
Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.
zipai-optimizer
Adaptive token optimizer: intelligent filtering, surgical output, ambiguity-first, context-window-aware, VCS-aware, MCP-aware.
zeroize-audit
Detects missing zeroization of sensitive data in source code and identifies zeroization removed by compiler optimizations, with assembly-level analysis, and control-flow verification. Use for auditing C/C++/Rust code handling secrets, keys, passwords, or other sensitive data.