gcp-spark
Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
- License 许可
- Apache-2.0
- 所属仓库
- saski/arnesto
- 最近更新
- 2026-11-12
- Stars
- ★ 6
- Forks
- 0
01 前言 Front Matter
- name
- gcp-spark
- license
- Apache-2.0
- metadata
- { "version": "v2", "publisher": "google" }
- description
- Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
02 资源文件 5
- 参考文档resources/gcloud_dataproc.md7.6 KB
- 参考文档resources/ml_tasks.md1.9 KB
- 参考文档resources/read_write_data.md13.3 KB
- 参考文档resources/schema_direct_inspection.md595 B
- 参考文档resources/spark_optimizations.md322 B
03 SKILL.md 正文
Spark on Dataproc
[!IMPORTANT] You MUST ALWAYS follow the Task Execution Workflow when writing spark code.
Task Execution Workflow
- Understand schemas: ALWAYS use
@skill:discovering-gcp-data-assetsskill orresources/schema_direct_inspection.mdto understand input and output schemas. Include the schema in your thought process BEFORE generating any code. Do NOT guess column names. - Generate spark code:
- Output Format: ALWAYS generate code in Python Notebooks (.ipynb) format. Generate scripts (.py) only if explicitly requested.
- Read and Write data: ALWAYS Refer to
resources/read_write_data.mdwhen reading or writing data. - ML Tasks: Refer to
@skill:ml-best-practicesskill andresources/ml_tasks.mdwhen generating ML code. - Spark Optimizations: ALWAYS refer to
resources/spark_optimizations.mdwhen generating spark code and apply optimization whenever applicable.
- Verify schema before write: ALWAYS verify that the dataframe and
destination schema match, use
df.printSchema()for dataframe schema and refer to@skill:discovering-gcp-data-assetsskill orresources/schema_direct_inspection.mdto verify destination schema. - Compile code before executing: For notebooks convert them to python
script using
jupyter nbconvert --to script your-notebook.ipynbfirst, then compile code usingpython3 -m py_compile your-notebook.py. - Execute script: ONLY when generating a
.pyscript refer toresources/gcloud_dataproc.mdon writing command to execute generated code on Dataproc. This DOES NOT apply when generating notebooks.
Common Mistakes Checklist
[!CAUTION] Ensure you verify this checklist to avoid mistakes
Before submitting a job, verify:
- [ ] All imports present (
col,when,lit, etc. frompyspark.sql.functions) - [ ]
vector_to_arrayfrom correct module usefrom pyspark.ml.functions import vector_to_array(NOTpyspark.sql.functions) - [ ] DataFrame schema matches target Iceberg table verify with
df.printSchema()before writing - [ ] CSV files read with
headerandinferSchemawithout these, the header row becomes data and all columns are strings - [ ] Avoid toPandas() Converting a pyspark dataframe to pandas by calling toPandas() can lead to out of memory errors. Only acceptable for building visualizations in Spark 3.5
IAM Requirements
The Dataproc service account needs:
roles/dataproc.worker: Job executionroles/biglake.admin: Iceberg table managementroles/bigquery.jobUser: Query materializationroles/storage.objectUser: Read/write GCSroles/spanner.databaseUser: Spanner writes
Spark resource management
Refer to resources/gcloud_dataproc.md for detailed guidelines on managing
Spark clusters, jobs, batches, and interactive sessions.
04 同仓库更多 Skills
visualize
Use /visualize for creating compact charts, diagrams, or other visuals inline as part of the conversation itself. It chooses among Markdown tables, Mermaid, inline HTML, and other supported in-transcript forms. Use Canvas for durable artifacts outside the transcript that users may revisit, refine, or share.
update-cursor-settings
Modify Cursor/VSCode user settings in settings.json. Use when you want to change editor settings, preferences, configuration, themes, font size, tab size, format on save, auto save, keybindings, or any settings.json values.
update-cli-config
View and modify Cursor CLI configuration settings in ~/.cursor/cli-config.json. Use when the user wants to change CLI settings, configure permissions, switch approval mode, enable vim mode, toggle display options, configure sandbox, or manage any CLI preferences.
statusline
Configure a custom status line in the CLI. Use when the user mentions status line, statusline, statusLine, CLI status bar, prompt footer customization, or wants to add session context above the prompt.
split-to-prs
Split current work into small reviewable PRs. Use when the user asks to split a chat, set of changes, branch, or PR.
shell
Runs the rest of a /shell request as a literal shell command. Use only when the user explicitly invokes /shell and wants the following text executed directly in the terminal.