databricks-core-workflow-a
Execute Databricks primary workflow: Delta Lake ETL pipelines. Use when building data ingestion pipelines, implementing medallion architecture, or creating Delta Lake transformations. Trigger with phrases like "databricks ETL", "delta lake pipeline", "medallion architecture", "databricks data pipeline", "bronze silver gold".
- License 许可
- MIT
- 最近更新
- 2026-05-26
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- databricks-core-workflow-a
- author
- Jeremy Longshore <jeremy@intentsolutions.io>
- license
- MIT
- version
- 1.0.0
- description
- Execute Databricks primary workflow: Delta Lake ETL pipelines. Use when building data ingestion pipelines, implementing medallion architecture, or creating Delta Lake transformations. Trigger with phrases like "databricks ETL", "delta lake pipeline", "medallion architecture", "databricks data pipeline", "bronze silver gold".
- allowed-tools
- Read, Write, Edit, Bash(databricks:*), Grep
- compatible-with
- claude-code, codex, openclaw
02 资源文件 1
- 参考文档references/implementation.md5.0 KB
03 SKILL.md 正文
Databricks Core Workflow A: Delta Lake ETL
Contents
Overview
Build production Delta Lake ETL pipelines using medallion architecture (Bronze -> Silver -> Gold).
Prerequisites
- Completed
databricks-install-authsetup - Understanding of Delta Lake concepts
- Unity Catalog configured (recommended)
Medallion Architecture
Raw Sources -> Bronze (Raw/Landing) -> Silver (Cleaned/Business Logic) -> Gold (Aggregated/Analytics Ready)
Instructions
Step 1: Bronze Layer - Raw Ingestion
Ingest raw data with metadata columns (_ingested_at, _source_file). Use mergeSchema for schema evolution. Use Auto Loader (cloudFiles) for streaming ingestion with schema inference.
Step 2: Silver Layer - Data Cleansing
Read from Bronze using Change Data Feed. Apply transformations: trim/lowercase strings, parse timestamps, hash PII, filter nulls, generate surrogate keys. Merge into Silver with upsert pattern.
Step 3: Gold Layer - Business Aggregations
Aggregate Silver data by business dimensions and time grain. Use partition-level overwrites for efficient updates.
Step 4: Delta Live Tables (Optional)
Declarative pipeline with @dlt.table decorators and data quality expectations (@dlt.expect_or_drop).
See detailed implementation for complete Bronze/Silver/Gold pipeline code, Auto Loader config, DLT pipeline, and orchestration example.
Output
- Bronze layer with raw data and ingestion metadata
- Silver layer with cleansed, deduplicated data
- Gold layer with business aggregations
- Delta Lake tables with ACID transactions
Error Handling
| Error | Cause | Solution |
|---|---|---|
| Schema mismatch | Source schema changed | Use mergeSchema option |
| Duplicate records | Missing deduplication | Add merge logic with primary keys |
| Null values | Data quality issues | Add expectations/filters in Silver |
| Memory errors | Large aggregations | Increase cluster size or partition data |
Examples
Quick Pipeline Run
# Full medallion pipeline
bronze.ingest_to_bronze(spark, "/mnt/landing/orders/", "catalog.bronze.orders")
silver.transform_to_silver(spark, "catalog.bronze.orders", "catalog.silver.orders", primary_keys=["order_id"])
gold.aggregate_to_gold(spark, "catalog.silver.orders", "catalog.gold.metrics", group_by_columns=["region"])
Resources
Next Steps
For ML workflows, see databricks-core-workflow-b.
04 同仓库更多 Skills
zustand-store-ts
Create Zustand stores with TypeScript, subscribeWithSelector middleware, and proper state/action separation. Use when building React state management, creating global stores, or implementing reactive state patterns with Zustand.
zustand-store-creator
Zustand Store Creator - Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category.
zero-trust-config-helper
Zero Trust Config Helper - Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category.
zapier-integration-helper
Zapier Integration Helper - Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category.
zai-cli
Execute z.AI CLI providing vision, search, reader, and GitHub exploration via CLI and MCP. Use when user needs image/video analysis, OCR, UI-to-code conversion, error diagnosis, real-time web search, web page to markdown extraction, or GitHub code exploration. Trigger with phrases like "analyze this image", "search the web for", "read this page", "explore this repo", or "use zai". Requires Z_AI_API_KEY.
yt-research
Research competitor YouTube channels, niches, and trending topics for your content strategy. Use this skill whenever the user says "research channels", "analyze competitors", "find trending topics", "niche analysis", "competitive research", "what are other creators doing", "scrape YouTube channels", or wants to understand the competitive landscape for a specific tool or topic area. Use when working with yt research. Trigger with 'yt', 'research'.