[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74WQQGZ82FKQ674M1Z2JSXJ":3,"skill-same:01M74WQQGZ82FKQ674M1Z2JSXJ":49},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":29,"commitSha":30,"skillDirectory":5,"forksCount":14,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":6,"updatedAt":34,"install":35,"resources":48},"01M74WQQGZ82FKQ674M1Z2JSXJ","databricks-reference-architecture",null,"Implement Databricks reference architecture with best-practice project layout.\nUse when designing new Databricks projects, reviewing architecture,\nor establishing standards for Databricks applications.\nTrigger with phrases like \"databricks architecture\", \"databricks best practices\",\n\"databricks project structure\", \"how to organize databricks\", \"databricks layout\".","SamuelChien\u002Fmega-skills-collection","SamuelChien","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F4652114?v=4","01M74KK34WVHA0P5ZYSJRDDFMG","Samuel Chien","01M74KK35FQN1N9VZ9EPTH3P2S",0,"MIT","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fdatabricks-reference-architecture\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: databricks-reference-architecture\ndescription: |\n  Implement Databricks reference architecture with best-practice project layout.\n  Use when designing new Databricks projects, reviewing architecture,\n  or establishing standards for Databricks applications.\n  Trigger with phrases like \"databricks architecture\", \"databricks best practices\",\n  \"databricks project structure\", \"how to organize databricks\", \"databricks layout\".\nallowed-tools: Read, Grep\nversion: 1.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\ncompatible-with: claude-code, codex, openclaw\n---\n# Databricks Reference Architecture\n\n## Overview\nProduction-ready lakehouse architecture with Unity Catalog, Delta Lake, and medallion pattern. Covers workspace organization, data governance, and CI\u002FCD integration for enterprise data platforms.\n\n## Prerequisites\n- Databricks workspace with Unity Catalog enabled\n- Understanding of medallion architecture (bronze\u002Fsilver\u002Fgold)\n- Databricks CLI configured\n- Terraform or Asset Bundles for infrastructure\n\n## Architecture Diagram\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Unity Catalog                             │\n│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │\n│  │ Bronze   │  │ Silver   │  │ Gold     │                   │\n│  │ Catalog  │─▶│ Catalog  │─▶│ Catalog  │                   │\n│  │ (raw)    │  │ (clean)  │  │ (curated)│                   │\n│  └──────────┘  └──────────┘  └──────────┘                   │\n│       ▲                            │                         │\n│       │                            ▼                         │\n│  ┌──────────┐              ┌──────────────┐                  │\n│  │ Ingestion│              │ ML Models    │                  │\n│  │ Jobs     │              │ (MLflow)     │                  │\n│  └──────────┘              └──────────────┘                  │\n├─────────────────────────────────────────────────────────────┤\n│  Compute: Job Clusters │ SQL Warehouses │ Interactive       │\n├─────────────────────────────────────────────────────────────┤\n│  Governance: Row-Level Security │ Column Masking │ Lineage  │\n└─────────────────────────────────────────────────────────────┘\n```\n\n## Project Structure\n\n```\ndatabricks-project\u002F\n├── src\u002F\n│   ├── ingestion\u002F\n│   │   ├── bronze_raw_events.py\n│   │   ├── bronze_api_data.py\n│   │   └── bronze_file_uploads.py\n│   ├── transformation\u002F\n│   │   ├── silver_clean_events.py\n│   │   ├── silver_deduplicate.py\n│   │   └── silver_schema_enforce.py\n│   ├── aggregation\u002F\n│   │   ├── gold_daily_metrics.py\n│   │   ├── gold_user_features.py\n│   │   └── gold_ml_features.py\n│   └── ml\u002F\n│       ├── training\u002F\n│       └── inference\u002F\n├── tests\u002F\n│   ├── unit\u002F\n│   └── integration\u002F\n├── databricks.yml           # Asset Bundle config\n├── resources\u002F\n│   ├── jobs.yml\n│   ├── pipelines.yml\n│   └── clusters.yml\n└── conf\u002F\n    ├── dev.yml\n    ├── staging.yml\n    └── prod.yml\n```\n\n## Instructions\n\n### Step 1: Configure Unity Catalog Hierarchy\n```sql\n-- Create catalog per environment\nCREATE CATALOG IF NOT EXISTS dev_catalog;\nCREATE CATALOG IF NOT EXISTS prod_catalog;\n\n-- Create schemas per medallion layer\nCREATE SCHEMA IF NOT EXISTS prod_catalog.bronze;\nCREATE SCHEMA IF NOT EXISTS prod_catalog.silver;\nCREATE SCHEMA IF NOT EXISTS prod_catalog.gold;\nCREATE SCHEMA IF NOT EXISTS prod_catalog.ml_features;\n\n-- Grant permissions\nGRANT USE CATALOG ON CATALOG prod_catalog TO `data-engineers`;\nGRANT USE SCHEMA ON SCHEMA prod_catalog.bronze TO `data-engineers`;\nGRANT SELECT ON SCHEMA prod_catalog.gold TO `analysts`;\n```\n\n### Step 2: Define Asset Bundle Configuration\n```yaml\n# databricks.yml\nbundle:\n  name: data-platform\n\nworkspace:\n  host: https:\u002F\u002Fyour-workspace.databricks.com\n\nresources:\n  jobs:\n    daily_etl:\n      name: \"Daily ETL Pipeline\"\n      schedule:\n        quartz_cron_expression: \"0 0 6 * * ?\"\n        timezone_id: \"UTC\"\n      tasks:\n        - task_key: bronze_ingest\n          notebook_task:\n            notebook_path: src\u002Fingestion\u002Fbronze_raw_events.py\n          job_cluster_key: etl_cluster\n        - task_key: silver_transform\n          depends_on:\n            - task_key: bronze_ingest\n          notebook_task:\n            notebook_path: src\u002Ftransformation\u002Fsilver_clean_events.py\n          job_cluster_key: etl_cluster\n\n  job_clusters:\n    - job_cluster_key: etl_cluster\n      new_cluster:\n        spark_version: \"14.3.x-scala2.12\"\n        node_type_id: \"i3.xlarge\"\n        num_workers: 2\n        autoscale:\n          min_workers: 1\n          max_workers: 4\n\ntargets:\n  dev:\n    default: true\n    workspace:\n      host: https:\u002F\u002Fdev-workspace.databricks.com\n  prod:\n    workspace:\n      host: https:\u002F\u002Fprod-workspace.databricks.com\n```\n\n### Step 3: Implement Medallion Pipeline Pattern\n```python\n# src\u002Fingestion\u002Fbronze_raw_events.py\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import current_timestamp, input_file_name\n\nspark = SparkSession.builder.getOrCreate()\n\n# Bronze: raw ingestion with metadata\nraw_df = (\n    spark.readStream\n    .format(\"cloudFiles\")\n    .option(\"cloudFiles.format\", \"json\")\n    .option(\"cloudFiles.schemaLocation\", \"\u002Fcheckpoints\u002Fbronze\u002Fschema\")\n    .load(\"\u002Fdata\u002Fraw\u002Fevents\u002F\")\n)\n\nbronze_df = raw_df.withColumn(\n    \"_ingested_at\", current_timestamp()\n).withColumn(\n    \"_source_file\", input_file_name()\n)\n\n(bronze_df.writeStream\n    .format(\"delta\")\n    .outputMode(\"append\")\n    .option(\"checkpointLocation\", \"\u002Fcheckpoints\u002Fbronze\u002Fevents\")\n    .toTable(\"prod_catalog.bronze.raw_events\"))\n```\n\n## Error Handling\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| Schema evolution failure | New columns in source | Enable `mergeSchema` option |\n| Job cluster timeout | Long-running tasks | Use autoscaling, increase timeout |\n| Permission denied | Missing catalog grants | Check Unity Catalog ACLs |\n| Delta version conflict | Concurrent writes | Use `MERGE` instead of `INSERT` |\n\n## Examples\n\n### Quick Medallion Validation\n```python\n# Validate data flows through medallion layers\nfor layer in [\"bronze\", \"silver\", \"gold\"]:\n    count = spark.table(f\"prod_catalog.{layer}.events\").count()\n    print(f\"{layer}: {count:,} rows\")\n```\n\n## Resources\n- [Databricks Asset Bundles](https:\u002F\u002Fdocs.databricks.com\u002Fdev-tools\u002Fbundles\u002F)\n- [Unity Catalog Best Practices](https:\u002F\u002Fdocs.databricks.com\u002Fdata-governance\u002Funity-catalog\u002F)\n- [Medallion Architecture](https:\u002F\u002Fwww.databricks.com\u002Fglossary\u002Fmedallion-architecture)\n\n## Output\n\n- Configuration files or code changes applied to the project\n- Validation report confirming correct implementation\n- Summary of changes made and their rationale","# Databricks Reference Architecture\n\n## Overview\nProduction-ready lakehouse architecture with Unity Catalog, Delta Lake, and medallion pattern. Covers workspace organization, data governance, and CI\u002FCD integration for enterprise data platforms.\n\n## Prerequisites\n- Databricks workspace with Unity Catalog enabled\n- Understanding of medallion architecture (bronze\u002Fsilver\u002Fgold)\n- Databricks CLI configured\n- Terraform or Asset Bundles for infrastructure\n\n## Architecture Diagram\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Unity Catalog                             │\n│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │\n│  │ Bronze   │  │ Silver   │  │ Gold     │                   │\n│  │ Catalog  │─▶│ Catalog  │─▶│ Catalog  │                   │\n│  │ (raw)    │  │ (clean)  │  │ (curated)│                   │\n│  └──────────┘  └──────────┘  └──────────┘                   │\n│       ▲                            │                         │\n│       │                            ▼                         │\n│  ┌──────────┐              ┌──────────────┐                  │\n│  │ Ingestion│              │ ML Models    │                  │\n│  │ Jobs     │              │ (MLflow)     │                  │\n│  └──────────┘              └──────────────┘                  │\n├─────────────────────────────────────────────────────────────┤\n│  Compute: Job Clusters │ SQL Warehouses │ Interactive       │\n├─────────────────────────────────────────────────────────────┤\n│  Governance: Row-Level Security │ Column Masking │ Lineage  │\n└─────────────────────────────────────────────────────────────┘\n```\n\n## Project Structure\n\n```\ndatabricks-project\u002F\n├── src\u002F\n│   ├── ingestion\u002F\n│   │   ├── bronze_raw_events.py\n│   │   ├── bronze_api_data.py\n│   │   └── bronze_file_uploads.py\n│   ├── transformation\u002F\n│   │   ├── silver_clean_events.py\n│   │   ├── silver_deduplicate.py\n│   │   └── silver_schema_enforce.py\n│   ├── aggregation\u002F\n│   │   ├── gold_daily_metrics.py\n│   │   ├── gold_user_features.py\n│   │   └── gold_ml_features.py\n│   └── ml\u002F\n│       ├── training\u002F\n│       └── inference\u002F\n├── tests\u002F\n│   ├── unit\u002F\n│   └── integration\u002F\n├── databricks.yml           # Asset Bundle config\n├── resources\u002F\n│   ├── jobs.yml\n│   ├── pipelines.yml\n│   └── clusters.yml\n└── conf\u002F\n    ├── dev.yml\n    ├── staging.yml\n    └── prod.yml\n```\n\n## Instructions\n\n### Step 1: Configure Unity Catalog Hierarchy\n```sql\n-- Create catalog per environment\nCREATE CATALOG IF NOT EXISTS dev_catalog;\nCREATE CATALOG IF NOT EXISTS prod_catalog;\n\n-- Create schemas per medallion layer\nCREATE SCHEMA IF NOT EXISTS prod_catalog.bronze;\nCREATE SCHEMA IF NOT EXISTS prod_catalog.silver;\nCREATE SCHEMA IF NOT EXISTS prod_catalog.gold;\nCREATE SCHEMA IF NOT EXISTS prod_catalog.ml_features;\n\n-- Grant permissions\nGRANT USE CATALOG ON CATALOG prod_catalog TO `data-engineers`;\nGRANT USE SCHEMA ON SCHEMA prod_catalog.bronze TO `data-engineers`;\nGRANT SELECT ON SCHEMA prod_catalog.gold TO `analysts`;\n```\n\n### Step 2: Define Asset Bundle Configuration\n```yaml\n# databricks.yml\nbundle:\n  name: data-platform\n\nworkspace:\n  host: https:\u002F\u002Fyour-workspace.databricks.com\n\nresources:\n  jobs:\n    daily_etl:\n      name: \"Daily ETL Pipeline\"\n      schedule:\n        quartz_cron_expression: \"0 0 6 * * ?\"\n        timezone_id: \"UTC\"\n      tasks:\n        - task_key: bronze_ingest\n          notebook_task:\n            notebook_path: src\u002Fingestion\u002Fbronze_raw_events.py\n          job_cluster_key: etl_cluster\n        - task_key: silver_transform\n          depends_on:\n            - task_key: bronze_ingest\n          notebook_task:\n            notebook_path: src\u002Ftransformation\u002Fsilver_clean_events.py\n          job_cluster_key: etl_cluster\n\n  job_clusters:\n    - job_cluster_key: etl_cluster\n      new_cluster:\n        spark_version: \"14.3.x-scala2.12\"\n        node_type_id: \"i3.xlarge\"\n        num_workers: 2\n        autoscale:\n          min_workers: 1\n          max_workers: 4\n\ntargets:\n  dev:\n    default: true\n    workspace:\n      host: https:\u002F\u002Fdev-workspace.databricks.com\n  prod:\n    workspace:\n      host: https:\u002F\u002Fprod-workspace.databricks.com\n```\n\n### Step 3: Implement Medallion Pipeline Pattern\n```python\n# src\u002Fingestion\u002Fbronze_raw_events.py\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import current_timestamp, input_file_name\n\nspark = SparkSession.builder.getOrCreate()\n\n# Bronze: raw ingestion with metadata\nraw_df = (\n    spark.readStream\n    .format(\"cloudFiles\")\n    .option(\"cloudFiles.format\", \"json\")\n    .option(\"cloudFiles.schemaLocation\", \"\u002Fcheckpoints\u002Fbronze\u002Fschema\")\n    .load(\"\u002Fdata\u002Fraw\u002Fevents\u002F\")\n)\n\nbronze_df = raw_df.withColumn(\n    \"_ingested_at\", current_timestamp()\n).withColumn(\n    \"_source_file\", input_file_name()\n)\n\n(bronze_df.writeStream\n    .format(\"delta\")\n    .outputMode(\"append\")\n    .option(\"checkpointLocation\", \"\u002Fcheckpoints\u002Fbronze\u002Fevents\")\n    .toTable(\"prod_catalog.bronze.raw_events\"))\n```\n\n## Error Handling\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| Schema evolution failure | New columns in source | Enable `mergeSchema` option |\n| Job cluster timeout | Long-running tasks | Use autoscaling, increase timeout |\n| Permission denied | Missing catalog grants | Check Unity Catalog ACLs |\n| Delta version conflict | Concurrent writes | Use `MERGE` instead of `INSERT` |\n\n## Examples\n\n### Quick Medallion Validation\n```python\n# Validate data flows through medallion layers\nfor layer in [\"bronze\", \"silver\", \"gold\"]:\n    count = spark.table(f\"prod_catalog.{layer}.events\").count()\n    print(f\"{layer}: {count:,} rows\")\n```\n\n## Resources\n- [Databricks Asset Bundles](https:\u002F\u002Fdocs.databricks.com\u002Fdev-tools\u002Fbundles\u002F)\n- [Unity Catalog Best Practices](https:\u002F\u002Fdocs.databricks.com\u002Fdata-governance\u002Funity-catalog\u002F)\n- [Medallion Architecture](https:\u002F\u002Fwww.databricks.com\u002Fglossary\u002Fmedallion-architecture)\n\n## Output\n\n- Configuration files or code changes applied to the project\n- Validation report confirming correct implementation\n- Summary of changes made and their rationale",{"name":5,"author":24,"license":15,"version":25,"description":26,"allowed-tools":27,"compatible-with":28},"Jeremy Longshore \u003Cjeremy@intentsolutions.io>","1.0.0","Implement Databricks reference architecture with best-practice project layout.\nUse when designing new Databricks projects, reviewing architecture,\nor establishing standards for Databricks applications.\nTrigger with phrases like \"databricks architecture\", \"databricks best practices\",\n\"databricks project structure\", \"how to organize databricks\", \"databricks layout\".\n","Read, Grep","claude-code, codex, openclaw","main","c2b1fdccc779e99ac814e55fb69ca8d71215dd0f","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection","https:\u002F\u002Fgithub.com\u002FSamuelChien","http:\u002F\u002Fsamuelchien.com\u002F","2026-05-26 08:27:14",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add SamuelChien\u002Fmega-skills-collection --skill databricks-reference-architecture",{"tool":41,"command":42},"bunx","bunx skills add SamuelChien\u002Fmega-skills-collection --skill databricks-reference-architecture",{"tool":44,"command":45},"pnpm","pnpm dlx skills add SamuelChien\u002Fmega-skills-collection --skill databricks-reference-architecture","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fdatabricks-reference-architecture\u002FSKILL.md",true,[],[50,57,64,71,78,86],{"skillNo":51,"name":52,"nameZh":6,"description":53,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":54,"categories":55,"keywords":56,"publishedAt":19},"01M76R4WZQZAXGGXMQCA0FZ716","zustand-store-ts","Create Zustand stores with TypeScript, subscribeWithSelector middleware, and proper state\u002Faction separation. Use when building React state management, creating global stores, or implementing reactive state patterns with Zustand.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":19},"01M76R4T9KENDH7FENGZQHBW4G","zustand-store-creator","Zustand Store Creator - Auto-activating skill for Frontend Development.\nTriggers on: zustand store creator, zustand store creator\nPart of the Frontend Development skill category.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzustand-store-creator\u002FSKILL.md",[],[],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M76R3YWN84B7SANJ52X799ZE","zero-trust-config-helper","Zero Trust Config Helper - Auto-activating skill for Security Advanced.\nTriggers on: zero trust config helper, zero trust config helper\nPart of the Security Advanced skill category.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzero-trust-config-helper\u002FSKILL.md",[],[],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M76R3HNSV2AEYSKHM915GJ58","zapier-integration-helper","Zapier Integration Helper - Auto-activating skill for Business Automation.\nTriggers on: zapier integration helper, zapier integration helper\nPart of the Business Automation skill category.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzapier-integration-helper\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M76R3DJSKAT4FA7DV4BF3GCE","zai-cli","Execute z.AI CLI providing vision, search, reader, and GitHub exploration via CLI and MCP.\nUse when user needs image\u002Fvideo analysis, OCR, UI-to-code conversion, error diagnosis,\nreal-time web search, web page to markdown extraction, or GitHub code exploration.\nTrigger with phrases like \"analyze this image\", \"search the web for\", \"read this page\",\n\"explore this repo\", or \"use zai\". Requires Z_AI_API_KEY.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzai-cli\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M76R2KAM699XE9AJVAWE4H2E","yt-research","Research competitor YouTube channels, niches, and trending topics for your content strategy.\nUse this skill whenever the user says \"research channels\", \"analyze competitors\", \"find trending topics\",\n\"niche analysis\", \"competitive research\", \"what are other creators doing\", \"scrape YouTube channels\",\nor wants to understand the competitive landscape for a specific tool or topic area. Use when working with yt research. Trigger with 'yt', 'research'.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fyt-research\u002FSKILL.md",[],[]]