gcp-spark
Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-21
- Stars
- ★ 3
- Forks
- 0
01 前言 Front Matter
- name
- gcp-spark
- license
- Apache-2.0
- metadata
- { "version": "v2", "publisher": "google" }
- description
- Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
02 资源文件 5
- 参考文档references/gcloud_dataproc.md7.6 KB
- 参考文档references/ml_tasks.md1.9 KB
- 参考文档references/read_write_data.md13.3 KB
- 参考文档references/schema_direct_inspection.md595 B
- 参考文档references/spark_optimizations.md322 B
03 SKILL.md 正文
Spark on Dataproc
[!IMPORTANT]
You MUST ALWAYS follow the Task Execution Workflow when writing spark code.
Task Execution Workflow
- Understand schemas: ALWAYS use
@skill:discovering-gcp-data-assetsskill orreferences/schema_direct_inspection.mdto understand input and output schemas. Include the schema in your thought process BEFORE generating any code. Do NOT guess column names. - Generate spark code:
- Output Format: ALWAYS generate code in Python Notebooks (.ipynb) format. Generate scripts (.py) only if explicitly requested.
- Read and Write data: ALWAYS Refer to
references/read_write_data.mdwhen reading or writing data. - ML Tasks: Refer to
@skill:ml-best-practicesskill andreferences/ml_tasks.mdwhen generating ML code. - Spark Optimizations: ALWAYS refer to
references/spark_optimizations.mdwhen generating spark code and apply optimization whenever applicable.
- Verify schema before write: ALWAYS verify that the dataframe and
destination schema match, use
df.printSchema()for dataframe schema and refer to@skill:discovering-gcp-data-assetsskill orreferences/schema_direct_inspection.mdto verify destination schema. - Compile code before executing: For notebooks convert them to python
script using
jupyter nbconvert --to script your-notebook.ipynbfirst, then compile code usingpython3 -m py_compile your-notebook.py. - Execute script: ONLY when generating a
.pyscript refer toreferences/gcloud_dataproc.mdon writing command to execute generated code on Dataproc. This DOES NOT apply when generating notebooks.
Common Mistakes Checklist
[!CAUTION]
Ensure you verify this checklist to avoid mistakes
Before submitting a job, verify:
- [ ] All imports present (
col,when,lit, etc. frompyspark.sql.functions) - [ ]
vector_to_arrayfrom correct module usefrom pyspark.ml.functions import vector_to_array(NOTpyspark.sql.functions) - [ ] DataFrame schema matches target Iceberg table verify with
df.printSchema()before writing - [ ] CSV files read with
headerandinferSchemawithout these, the header row becomes data and all columns are strings - [ ] Avoid toPandas() Converting a pyspark dataframe to pandas by calling toPandas() can lead to out of memory errors. Only acceptable for building visualizations in Spark 3.5
IAM Requirements
The Dataproc service account needs:
roles/dataproc.worker: Job executionroles/biglake.admin: Iceberg table managementroles/bigquery.jobUser: Query materializationroles/storage.objectUser: Read/write GCSroles/spanner.databaseUser: Spanner writes
Spark resource management
Refer to references/gcloud_dataproc.md for detailed guidelines on managing
Spark clusters, jobs, batches, and interactive sessions.
04 同仓库更多 Skills
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
web-artifacts-builder
Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn/ui). Use for complex artifacts requiring state management, routing, or shadcn/ui components - not for simple single-file HTML/JSX artifacts.
ui-styling
Create beautiful, accessible user interfaces with shadcn/ui components (built on Radix UI + Tailwind), Tailwind CSS utility-first styling, and canvas-based visual designs. Use when building user interfaces, implementing design systems, creating responsive layouts, adding accessible components (dialogs, dropdowns, forms, tables), customizing themes and colors, implementing dark mode, generating visual designs and posters, or establishing consistent styling patterns across applications.
design
Comprehensive design skill: brand identity, design tokens, UI styling, logo generation (55 styles, Gemini AI), corporate identity program (50 deliverables, CIP mockups), HTML presentations (Chart.js), banner design (22 styles, social/ads/web/print), icon design (15 styles, SVG, Gemini 3.1 Pro), social photos (HTML→screenshot, multi-platform). Actions: design logo, create CIP, generate mockups, build slides, design banner, generate icon, create social photos, social media images, brand identity, design system. Platforms: Facebook, Twitter, LinkedIn, YouTube, Instagram, Pinterest, TikTok, Threads, Google Ads.
design-system
Token architecture, component specifications, and slide generation. Three-layer tokens (primitive→semantic→component), CSS variables, spacing/typography scales, component specs, strategic slide creation. Use for design tokens, systematic design, brand-compliant presentations.
banner-design
Design banners for social media, ads, website heroes, creative assets, and print. Multiple art direction options with AI-generated visuals. Actions: design, create, generate banner. Platforms: Facebook, Twitter/X, LinkedIn, YouTube, Instagram, Google Display, website hero, print. Styles: minimalist, gradient, bold typography, photo-based, illustrated, geometric, retro, glassmorphism, 3D, neon, duotone, editorial, collage. Uses ui-ux-pro-max, frontend-design, ai-artist, ai-multimodal skills.