AGENT SKILL

databricks-reference-architecture

Implement Databricks reference architecture with best-practice project layout. Use when designing new Databricks projects, reviewing architecture, or establishing standards for Databricks applications. Trigger with phrases like "databricks architecture", "databricks best practices", "databricks project structure", "how to organize databricks", "databricks layout".

License 许可
MIT
最近更新
2026-05-26
Stars
★ 0
Forks
0

01 前言 Front Matter

name
databricks-reference-architecture
author
Jeremy Longshore <jeremy@intentsolutions.io>
license
MIT
version
1.0.0
description
Implement Databricks reference architecture with best-practice project layout. Use when designing new Databricks projects, reviewing architecture, or establishing standards for Databricks applications. Trigger with phrases like "databricks architecture", "databricks best practices", "databricks project structure", "how to organize databricks", "databricks layout".
allowed-tools
Read, Grep
compatible-with
claude-code, codex, openclaw

03 SKILL.md 正文

Databricks Reference Architecture

Overview

Production-ready lakehouse architecture with Unity Catalog, Delta Lake, and medallion pattern. Covers workspace organization, data governance, and CI/CD integration for enterprise data platforms.

Prerequisites

  • Databricks workspace with Unity Catalog enabled
  • Understanding of medallion architecture (bronze/silver/gold)
  • Databricks CLI configured
  • Terraform or Asset Bundles for infrastructure

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                    Unity Catalog                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │
│  │ Bronze   │  │ Silver   │  │ Gold     │                   │
│  │ Catalog  │─▶│ Catalog  │─▶│ Catalog  │                   │
│  │ (raw)    │  │ (clean)  │  │ (curated)│                   │
│  └──────────┘  └──────────┘  └──────────┘                   │
│       ▲                            │                         │
│       │                            ▼                         │
│  ┌──────────┐              ┌──────────────┐                  │
│  │ Ingestion│              │ ML Models    │                  │
│  │ Jobs     │              │ (MLflow)     │                  │
│  └──────────┘              └──────────────┘                  │
├─────────────────────────────────────────────────────────────┤
│  Compute: Job Clusters │ SQL Warehouses │ Interactive       │
├─────────────────────────────────────────────────────────────┤
│  Governance: Row-Level Security │ Column Masking │ Lineage  │
└─────────────────────────────────────────────────────────────┘

Project Structure

databricks-project/
├── src/
│   ├── ingestion/
│   │   ├── bronze_raw_events.py
│   │   ├── bronze_api_data.py
│   │   └── bronze_file_uploads.py
│   ├── transformation/
│   │   ├── silver_clean_events.py
│   │   ├── silver_deduplicate.py
│   │   └── silver_schema_enforce.py
│   ├── aggregation/
│   │   ├── gold_daily_metrics.py
│   │   ├── gold_user_features.py
│   │   └── gold_ml_features.py
│   └── ml/
│       ├── training/
│       └── inference/
├── tests/
│   ├── unit/
│   └── integration/
├── databricks.yml           # Asset Bundle config
├── resources/
│   ├── jobs.yml
│   ├── pipelines.yml
│   └── clusters.yml
└── conf/
    ├── dev.yml
    ├── staging.yml
    └── prod.yml

Instructions

Step 1: Configure Unity Catalog Hierarchy

-- Create catalog per environment
CREATE CATALOG IF NOT EXISTS dev_catalog;
CREATE CATALOG IF NOT EXISTS prod_catalog;

-- Create schemas per medallion layer
CREATE SCHEMA IF NOT EXISTS prod_catalog.bronze;
CREATE SCHEMA IF NOT EXISTS prod_catalog.silver;
CREATE SCHEMA IF NOT EXISTS prod_catalog.gold;
CREATE SCHEMA IF NOT EXISTS prod_catalog.ml_features;

-- Grant permissions
GRANT USE CATALOG ON CATALOG prod_catalog TO `data-engineers`;
GRANT USE SCHEMA ON SCHEMA prod_catalog.bronze TO `data-engineers`;
GRANT SELECT ON SCHEMA prod_catalog.gold TO `analysts`;

Step 2: Define Asset Bundle Configuration

# databricks.yml
bundle:
  name: data-platform

workspace:
  host: https://your-workspace.databricks.com

resources:
  jobs:
    daily_etl:
      name: "Daily ETL Pipeline"
      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "UTC"
      tasks:
        - task_key: bronze_ingest
          notebook_task:
            notebook_path: src/ingestion/bronze_raw_events.py
          job_cluster_key: etl_cluster
        - task_key: silver_transform
          depends_on:
            - task_key: bronze_ingest
          notebook_task:
            notebook_path: src/transformation/silver_clean_events.py
          job_cluster_key: etl_cluster

  job_clusters:
    - job_cluster_key: etl_cluster
      new_cluster:
        spark_version: "14.3.x-scala2.12"
        node_type_id: "i3.xlarge"
        num_workers: 2
        autoscale:
          min_workers: 1
          max_workers: 4

targets:
  dev:
    default: true
    workspace:
      host: https://dev-workspace.databricks.com
  prod:
    workspace:
      host: https://prod-workspace.databricks.com

Step 3: Implement Medallion Pipeline Pattern

# src/ingestion/bronze_raw_events.py
from pyspark.sql import SparkSession
from pyspark.sql.functions import current_timestamp, input_file_name

spark = SparkSession.builder.getOrCreate()

# Bronze: raw ingestion with metadata
raw_df = (
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/checkpoints/bronze/schema")
    .load("/data/raw/events/")
)

bronze_df = raw_df.withColumn(
    "_ingested_at", current_timestamp()
).withColumn(
    "_source_file", input_file_name()
)

(bronze_df.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "/checkpoints/bronze/events")
    .toTable("prod_catalog.bronze.raw_events"))

Error Handling

Issue Cause Solution
Schema evolution failure New columns in source Enable mergeSchema option
Job cluster timeout Long-running tasks Use autoscaling, increase timeout
Permission denied Missing catalog grants Check Unity Catalog ACLs
Delta version conflict Concurrent writes Use MERGE instead of INSERT

Examples

Quick Medallion Validation

# Validate data flows through medallion layers
for layer in ["bronze", "silver", "gold"]:
    count = spark.table(f"prod_catalog.{layer}.events").count()
    print(f"{layer}: {count:,} rows")

Resources

Output

  • Configuration files or code changes applied to the project
  • Validation report confirming correct implementation
  • Summary of changes made and their rationale

04 同仓库更多 Skills

SamuelChien
SamuelChien/mega-skills-collection

zustand-store-ts

Create Zustand stores with TypeScript, subscribeWithSelector middleware, and proper state/action separation. Use when building React state management, creating global stores, or implementing reactive state patterns with Zustand.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zustand-store-creator

Zustand Store Creator - Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zero-trust-config-helper

Zero Trust Config Helper - Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zapier-integration-helper

Zapier Integration Helper - Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zai-cli

Execute z.AI CLI providing vision, search, reader, and GitHub exploration via CLI and MCP. Use when user needs image/video analysis, OCR, UI-to-code conversion, error diagnosis, real-time web search, web page to markdown extraction, or GitHub code exploration. Trigger with phrases like "analyze this image", "search the web for", "read this page", "explore this repo", or "use zai". Requires Z_AI_API_KEY.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

yt-research

Research competitor YouTube channels, niches, and trending topics for your content strategy. Use this skill whenever the user says "research channels", "analyze competitors", "find trending topics", "niche analysis", "competitive research", "what are other creators doing", "scrape YouTube channels", or wants to understand the competitive landscape for a specific tool or topic area. Use when working with yt research. Trigger with 'yt', 'research'.

★ 0