[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6F67TAQMN3Z9QYYRSCSPS3R":3,"skill-same:01M6F67TAQMN3Z9QYYRSCSPS3R":73},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":53,"repositoryHtmlUrl":54,"repositoryHomepageUrl":6,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M6F67TAQMN3Z9QYYRSCSPS3R","databricks-performance-tuning",null,"Optimize Databricks cluster and query performance.\nUse when jobs are running slowly, optimizing Spark configurations,\nor improving Delta Lake query performance.\nTrigger with phrases like \"databricks performance\", \"spark tuning\",\n\"databricks slow\", \"optimize databricks\", \"cluster performance\".","Dicklesworthstone\u002Fpi_agent_rust","Dicklesworthstone","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F35050222?v=4","01M6CG26AR942CSVMN1M5E3DSE","Jeff Emanuel","01M6CG26EPP71PSHW1VDS71121",1981,"MIT","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Fplugins-community\u002Fplugins\u002Fsaas-packs\u002Fdatabricks-pack\u002Fskills\u002Fdatabricks-performance-tuning\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"databricks","performance","tuning","databricks performance tuning","performance tuning","running","when","with","cluster","trigger","io","query","optimize","delta","use","edit","write","read","grep","Bash","delta lake","spark","2026-11-10 10:12:30",false,"---\nname: databricks-performance-tuning\ndescription: |\n  Optimize Databricks cluster and query performance.\n  Use when jobs are running slowly, optimizing Spark configurations,\n  or improving Delta Lake query performance.\n  Trigger with phrases like \"databricks performance\", \"spark tuning\",\n  \"databricks slow\", \"optimize databricks\", \"cluster performance\".\nallowed-tools: Read, Write, Edit, Bash(databricks:*), Grep\nversion: 1.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\n---\n\n# Databricks Performance Tuning\n\n## Overview\nOptimize Databricks cluster, Spark, and Delta Lake performance.\n\n## Prerequisites\n- Access to cluster configuration\n- Understanding of workload characteristics\n- Query history access\n\n## Instructions\n\n### Step 1: Cluster Sizing\n\n```python\n# Cluster sizing calculator\ndef recommend_cluster_size(\n    data_size_gb: float,\n    complexity: str = \"medium\",  # low, medium, high\n    parallelism_need: str = \"standard\",  # standard, high\n) -> dict:\n    \"\"\"\n    Recommend cluster configuration based on workload.\n\n    Args:\n        data_size_gb: Estimated data size to process\n        complexity: Query\u002Ftransform complexity\n        parallelism_need: Required parallelism level\n\n    Returns:\n        Recommended cluster configuration\n    \"\"\"\n    # Memory per executor (standard DS3_v2 = 14GB)\n    memory_per_worker = 14\n\n    # Base calculation\n    workers_by_data = max(1, int(data_size_gb \u002F memory_per_worker \u002F 2))\n\n    # Adjust for complexity\n    complexity_multiplier = {\"low\": 1, \"medium\": 1.5, \"high\": 2.5}\n    workers = int(workers_by_data * complexity_multiplier.get(complexity, 1.5))\n\n    # Adjust for parallelism\n    if parallelism_need == \"high\":\n        workers = max(workers, 8)\n\n    return {\n        \"node_type_id\": \"Standard_DS3_v2\",\n        \"num_workers\": workers,\n        \"autoscale\": {\n            \"min_workers\": max(1, workers \u002F\u002F 2),\n            \"max_workers\": workers * 2,\n        },\n        \"spark_conf\": {\n            \"spark.sql.shuffle.partitions\": str(workers * 4),\n            \"spark.default.parallelism\": str(workers * 4),\n        }\n    }\n```\n\n### Step 2: Spark Configuration Optimization\n\n```python\n# Optimized Spark configurations by workload type\nspark_configs = {\n    \"etl_batch\": {\n        # Memory and parallelism\n        \"spark.sql.shuffle.partitions\": \"200\",\n        \"spark.default.parallelism\": \"200\",\n        \"spark.sql.files.maxPartitionBytes\": \"134217728\",  # 128MB\n\n        # Delta Lake optimizations\n        \"spark.databricks.delta.optimizeWrite.enabled\": \"true\",\n        \"spark.databricks.delta.autoCompact.enabled\": \"true\",\n        \"spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite\": \"true\",\n\n        # Adaptive query execution\n        \"spark.sql.adaptive.enabled\": \"true\",\n        \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n        \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n    },\n\n    \"ml_training\": {\n        # Memory for ML workloads\n        \"spark.driver.memory\": \"16g\",\n        \"spark.executor.memory\": \"16g\",\n        \"spark.memory.fraction\": \"0.8\",\n        \"spark.memory.storageFraction\": \"0.3\",\n\n        # Serialization for ML\n        \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n        \"spark.kryoserializer.buffer.max\": \"1024m\",\n    },\n\n    \"streaming\": {\n        # Streaming configurations\n        \"spark.sql.streaming.schemaInference\": \"true\",\n        \"spark.sql.streaming.checkpointLocation\": \"\u002Fmnt\u002Fcheckpoints\",\n        \"spark.databricks.delta.autoCompact.minNumFiles\": \"10\",\n\n        # Micro-batch tuning\n        \"spark.sql.streaming.forEachBatch.enabled\": \"true\",\n    },\n\n    \"interactive\": {\n        # Fast startup\n        \"spark.databricks.cluster.profile\": \"singleNode\",\n        \"spark.master\": \"local[*]\",\n\n        # Caching\n        \"spark.sql.inMemoryColumnarStorage.compressed\": \"true\",\n        \"spark.sql.inMemoryColumnarStorage.batchSize\": \"10000\",\n    }\n}\n```\n\n### Step 3: Delta Lake Optimization\n\n```python\n# delta_optimization.py\nfrom pyspark.sql import SparkSession\n\ndef optimize_delta_table(\n    spark: SparkSession,\n    table_name: str,\n    z_order_columns: list[str] = None,\n    vacuum_hours: int = 168,  # 7 days\n) -> dict:\n    \"\"\"\n    Optimize Delta table for query performance.\n\n    Args:\n        spark: SparkSession\n        table_name: Fully qualified table name\n        z_order_columns: Columns for Z-ordering (max 4)\n        vacuum_hours: Retention period for vacuum\n\n    Returns:\n        Optimization results\n    \"\"\"\n    results = {}\n\n    # 1. Run OPTIMIZE with Z-ordering\n    if z_order_columns:\n        z_order_clause = \", \".join(z_order_columns[:4])  # Max 4 columns\n        spark.sql(f\"OPTIMIZE {table_name} ZORDER BY ({z_order_clause})\")\n        results[\"z_order\"] = z_order_columns\n    else:\n        spark.sql(f\"OPTIMIZE {table_name}\")\n\n    results[\"optimized\"] = True\n\n    # 2. Analyze table statistics\n    spark.sql(f\"ANALYZE TABLE {table_name} COMPUTE STATISTICS\")\n    results[\"statistics_computed\"] = True\n\n    # 3. Vacuum old files\n    spark.sql(f\"VACUUM {table_name} RETAIN {vacuum_hours} HOURS\")\n    results[\"vacuumed\"] = True\n\n    # 4. Get table metrics\n    detail = spark.sql(f\"DESCRIBE DETAIL {table_name}\").first()\n    results[\"metrics\"] = {\n        \"num_files\": detail.numFiles,\n        \"size_bytes\": detail.sizeInBytes,\n        \"partitions\": detail.partitionColumns,\n    }\n\n    return results\n\ndef enable_liquid_clustering(\n    spark: SparkSession,\n    table_name: str,\n    cluster_columns: list[str],\n) -> None:\n    \"\"\"\n    Enable Liquid Clustering for automatic data layout optimization.\n\n    Liquid Clustering replaces traditional partitioning and Z-ordering\n    with automatic, incremental clustering.\n    \"\"\"\n    columns = \", \".join(cluster_columns)\n    spark.sql(f\"\"\"\n        ALTER TABLE {table_name}\n        CLUSTER BY ({columns})\n    \"\"\")\n\ndef enable_predictive_optimization(\n    spark: SparkSession,\n    table_name: str,\n) -> None:\n    \"\"\"Enable Databricks Predictive Optimization.\"\"\"\n    spark.sql(f\"\"\"\n        ALTER TABLE {table_name}\n        SET TBLPROPERTIES (\n            'delta.enableDeletionVectors' = 'true',\n            'delta.targetFileSize' = '104857600'\n        )\n    \"\"\")\n```\n\n### Step 4: Query Performance Analysis\n\n```sql\n-- Find slow queries from query history\nSELECT\n    query_id,\n    query_text,\n    duration \u002F 1000 as seconds,\n    rows_produced,\n    bytes_read,\n    start_time\nFROM system.query.history\nWHERE duration > 60000  -- > 60 seconds\n  AND start_time > current_timestamp() - INTERVAL 24 HOURS\nORDER BY duration DESC\nLIMIT 20;\n\n-- Analyze query plan\nEXPLAIN FORMATTED\nSELECT * FROM main.silver.orders\nWHERE order_date > '2024-01-01'\n  AND region = 'US';\n\n-- Check table scan statistics\nSELECT\n    table_name,\n    SUM(bytes_read) \u002F 1024 \u002F 1024 \u002F 1024 as gb_read,\n    SUM(rows_produced) as total_rows,\n    COUNT(*) as query_count\nFROM system.query.history\nWHERE start_time > current_timestamp() - INTERVAL 7 DAYS\nGROUP BY table_name\nORDER BY gb_read DESC;\n```\n\n### Step 5: Caching Strategy\n\n```python\n# Intelligent caching for repeated queries\nfrom pyspark.sql import DataFrame\nfrom functools import lru_cache\n\nclass CacheManager:\n    \"\"\"Manage Spark DataFrame caching.\"\"\"\n\n    def __init__(self, spark: SparkSession):\n        self.spark = spark\n        self._cache_registry = {}\n\n    def cache_table(\n        self,\n        table_name: str,\n        cache_level: str = \"MEMORY_AND_DISK\",\n    ) -> DataFrame:\n        \"\"\"Cache table with specified storage level.\"\"\"\n        if table_name in self._cache_registry:\n            return self._cache_registry[table_name]\n\n        df = self.spark.table(table_name)\n\n        if cache_level == \"MEMORY_ONLY\":\n            df.cache()\n        elif cache_level == \"MEMORY_AND_DISK\":\n            from pyspark import StorageLevel\n            df.persist(StorageLevel.MEMORY_AND_DISK)\n        elif cache_level == \"DISK_ONLY\":\n            from pyspark import StorageLevel\n            df.persist(StorageLevel.DISK_ONLY)\n\n        # Trigger caching\n        df.count()\n\n        self._cache_registry[table_name] = df\n        return df\n\n    def uncache_all(self):\n        \"\"\"Clear all cached DataFrames.\"\"\"\n        for df in self._cache_registry.values():\n            df.unpersist()\n        self._cache_registry.clear()\n        self.spark.catalog.clearCache()\n\n# Delta Cache (automatic)\n# Enable in cluster config:\n# \"spark.databricks.io.cache.enabled\": \"true\"\n# \"spark.databricks.io.cache.maxDiskUsage\": \"50g\"\n```\n\n### Step 6: Join Optimization\n\n```python\nfrom pyspark.sql import DataFrame\nfrom pyspark.sql.functions import broadcast\n\ndef optimize_join(\n    df_large: DataFrame,\n    df_small: DataFrame,\n    join_key: str,\n    small_table_threshold_mb: int = 100,\n) -> DataFrame:\n    \"\"\"\n    Optimize join based on table sizes.\n\n    Uses broadcast join for small tables,\n    sort-merge join for large tables.\n    \"\"\"\n    # Estimate small table size\n    small_size_mb = df_small.count() * 100 \u002F 1024 \u002F 1024  # rough estimate\n\n    if small_size_mb \u003C small_table_threshold_mb:\n        # Broadcast join (small table fits in memory)\n        return df_large.join(broadcast(df_small), join_key)\n    else:\n        # Sort-merge join with bucketing hint\n        return df_large.join(df_small, join_key, \"inner\")\n\n# Bucketed tables for frequent joins\ndef create_bucketed_table(\n    spark: SparkSession,\n    df: DataFrame,\n    table_name: str,\n    bucket_columns: list[str],\n    num_buckets: int = 100,\n):\n    \"\"\"Create bucketed table for join optimization.\"\"\"\n    (\n        df.write\n        .bucketBy(num_buckets, *bucket_columns)\n        .sortBy(*bucket_columns)\n        .saveAsTable(table_name)\n    )\n```\n\n## Output\n- Optimized cluster configuration\n- Tuned Spark settings\n- Optimized Delta tables\n- Improved query performance\n\n## Error Handling\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| OOM errors | Insufficient memory | Increase executor memory or reduce partition size |\n| Skewed data | Uneven distribution | Use salting or AQE skew handling |\n| Slow joins | Large shuffle | Use broadcast for small tables |\n| Too many files | Small files problem | Run OPTIMIZE regularly |\n\n## Examples\n\n### Performance Benchmark\n```python\nimport time\n\ndef benchmark_query(spark, query: str, runs: int = 3) -> dict:\n    \"\"\"Benchmark query execution time.\"\"\"\n    times = []\n    for _ in range(runs):\n        spark.catalog.clearCache()\n        start = time.time()\n        spark.sql(query).collect()\n        times.append(time.time() - start)\n\n    return {\n        \"min\": min(times),\n        \"max\": max(times),\n        \"avg\": sum(times) \u002F len(times),\n        \"runs\": runs,\n    }\n```\n\n## Resources\n- [Databricks Performance Guide](https:\u002F\u002Fdocs.databricks.com\u002Foptimizations\u002Findex.html)\n- [Delta Lake Optimization](https:\u002F\u002Fdocs.databricks.com\u002Fdelta\u002Foptimizations\u002Findex.html)\n- [Spark Tuning Guide](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Ftuning.html)\n\n## Next Steps\nFor cost optimization, see `databricks-cost-tuning`.\n","# Databricks Performance Tuning\n\n## Overview\nOptimize Databricks cluster, Spark, and Delta Lake performance.\n\n## Prerequisites\n- Access to cluster configuration\n- Understanding of workload characteristics\n- Query history access\n\n## Instructions\n\n### Step 1: Cluster Sizing\n\n```python\n# Cluster sizing calculator\ndef recommend_cluster_size(\n    data_size_gb: float,\n    complexity: str = \"medium\",  # low, medium, high\n    parallelism_need: str = \"standard\",  # standard, high\n) -> dict:\n    \"\"\"\n    Recommend cluster configuration based on workload.\n\n    Args:\n        data_size_gb: Estimated data size to process\n        complexity: Query\u002Ftransform complexity\n        parallelism_need: Required parallelism level\n\n    Returns:\n        Recommended cluster configuration\n    \"\"\"\n    # Memory per executor (standard DS3_v2 = 14GB)\n    memory_per_worker = 14\n\n    # Base calculation\n    workers_by_data = max(1, int(data_size_gb \u002F memory_per_worker \u002F 2))\n\n    # Adjust for complexity\n    complexity_multiplier = {\"low\": 1, \"medium\": 1.5, \"high\": 2.5}\n    workers = int(workers_by_data * complexity_multiplier.get(complexity, 1.5))\n\n    # Adjust for parallelism\n    if parallelism_need == \"high\":\n        workers = max(workers, 8)\n\n    return {\n        \"node_type_id\": \"Standard_DS3_v2\",\n        \"num_workers\": workers,\n        \"autoscale\": {\n            \"min_workers\": max(1, workers \u002F\u002F 2),\n            \"max_workers\": workers * 2,\n        },\n        \"spark_conf\": {\n            \"spark.sql.shuffle.partitions\": str(workers * 4),\n            \"spark.default.parallelism\": str(workers * 4),\n        }\n    }\n```\n\n### Step 2: Spark Configuration Optimization\n\n```python\n# Optimized Spark configurations by workload type\nspark_configs = {\n    \"etl_batch\": {\n        # Memory and parallelism\n        \"spark.sql.shuffle.partitions\": \"200\",\n        \"spark.default.parallelism\": \"200\",\n        \"spark.sql.files.maxPartitionBytes\": \"134217728\",  # 128MB\n\n        # Delta Lake optimizations\n        \"spark.databricks.delta.optimizeWrite.enabled\": \"true\",\n        \"spark.databricks.delta.autoCompact.enabled\": \"true\",\n        \"spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite\": \"true\",\n\n        # Adaptive query execution\n        \"spark.sql.adaptive.enabled\": \"true\",\n        \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n        \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n    },\n\n    \"ml_training\": {\n        # Memory for ML workloads\n        \"spark.driver.memory\": \"16g\",\n        \"spark.executor.memory\": \"16g\",\n        \"spark.memory.fraction\": \"0.8\",\n        \"spark.memory.storageFraction\": \"0.3\",\n\n        # Serialization for ML\n        \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n        \"spark.kryoserializer.buffer.max\": \"1024m\",\n    },\n\n    \"streaming\": {\n        # Streaming configurations\n        \"spark.sql.streaming.schemaInference\": \"true\",\n        \"spark.sql.streaming.checkpointLocation\": \"\u002Fmnt\u002Fcheckpoints\",\n        \"spark.databricks.delta.autoCompact.minNumFiles\": \"10\",\n\n        # Micro-batch tuning\n        \"spark.sql.streaming.forEachBatch.enabled\": \"true\",\n    },\n\n    \"interactive\": {\n        # Fast startup\n        \"spark.databricks.cluster.profile\": \"singleNode\",\n        \"spark.master\": \"local[*]\",\n\n        # Caching\n        \"spark.sql.inMemoryColumnarStorage.compressed\": \"true\",\n        \"spark.sql.inMemoryColumnarStorage.batchSize\": \"10000\",\n    }\n}\n```\n\n### Step 3: Delta Lake Optimization\n\n```python\n# delta_optimization.py\nfrom pyspark.sql import SparkSession\n\ndef optimize_delta_table(\n    spark: SparkSession,\n    table_name: str,\n    z_order_columns: list[str] = None,\n    vacuum_hours: int = 168,  # 7 days\n) -> dict:\n    \"\"\"\n    Optimize Delta table for query performance.\n\n    Args:\n        spark: SparkSession\n        table_name: Fully qualified table name\n        z_order_columns: Columns for Z-ordering (max 4)\n        vacuum_hours: Retention period for vacuum\n\n    Returns:\n        Optimization results\n    \"\"\"\n    results = {}\n\n    # 1. Run OPTIMIZE with Z-ordering\n    if z_order_columns:\n        z_order_clause = \", \".join(z_order_columns[:4])  # Max 4 columns\n        spark.sql(f\"OPTIMIZE {table_name} ZORDER BY ({z_order_clause})\")\n        results[\"z_order\"] = z_order_columns\n    else:\n        spark.sql(f\"OPTIMIZE {table_name}\")\n\n    results[\"optimized\"] = True\n\n    # 2. Analyze table statistics\n    spark.sql(f\"ANALYZE TABLE {table_name} COMPUTE STATISTICS\")\n    results[\"statistics_computed\"] = True\n\n    # 3. Vacuum old files\n    spark.sql(f\"VACUUM {table_name} RETAIN {vacuum_hours} HOURS\")\n    results[\"vacuumed\"] = True\n\n    # 4. Get table metrics\n    detail = spark.sql(f\"DESCRIBE DETAIL {table_name}\").first()\n    results[\"metrics\"] = {\n        \"num_files\": detail.numFiles,\n        \"size_bytes\": detail.sizeInBytes,\n        \"partitions\": detail.partitionColumns,\n    }\n\n    return results\n\ndef enable_liquid_clustering(\n    spark: SparkSession,\n    table_name: str,\n    cluster_columns: list[str],\n) -> None:\n    \"\"\"\n    Enable Liquid Clustering for automatic data layout optimization.\n\n    Liquid Clustering replaces traditional partitioning and Z-ordering\n    with automatic, incremental clustering.\n    \"\"\"\n    columns = \", \".join(cluster_columns)\n    spark.sql(f\"\"\"\n        ALTER TABLE {table_name}\n        CLUSTER BY ({columns})\n    \"\"\")\n\ndef enable_predictive_optimization(\n    spark: SparkSession,\n    table_name: str,\n) -> None:\n    \"\"\"Enable Databricks Predictive Optimization.\"\"\"\n    spark.sql(f\"\"\"\n        ALTER TABLE {table_name}\n        SET TBLPROPERTIES (\n            'delta.enableDeletionVectors' = 'true',\n            'delta.targetFileSize' = '104857600'\n        )\n    \"\"\")\n```\n\n### Step 4: Query Performance Analysis\n\n```sql\n-- Find slow queries from query history\nSELECT\n    query_id,\n    query_text,\n    duration \u002F 1000 as seconds,\n    rows_produced,\n    bytes_read,\n    start_time\nFROM system.query.history\nWHERE duration > 60000  -- > 60 seconds\n  AND start_time > current_timestamp() - INTERVAL 24 HOURS\nORDER BY duration DESC\nLIMIT 20;\n\n-- Analyze query plan\nEXPLAIN FORMATTED\nSELECT * FROM main.silver.orders\nWHERE order_date > '2024-01-01'\n  AND region = 'US';\n\n-- Check table scan statistics\nSELECT\n    table_name,\n    SUM(bytes_read) \u002F 1024 \u002F 1024 \u002F 1024 as gb_read,\n    SUM(rows_produced) as total_rows,\n    COUNT(*) as query_count\nFROM system.query.history\nWHERE start_time > current_timestamp() - INTERVAL 7 DAYS\nGROUP BY table_name\nORDER BY gb_read DESC;\n```\n\n### Step 5: Caching Strategy\n\n```python\n# Intelligent caching for repeated queries\nfrom pyspark.sql import DataFrame\nfrom functools import lru_cache\n\nclass CacheManager:\n    \"\"\"Manage Spark DataFrame caching.\"\"\"\n\n    def __init__(self, spark: SparkSession):\n        self.spark = spark\n        self._cache_registry = {}\n\n    def cache_table(\n        self,\n        table_name: str,\n        cache_level: str = \"MEMORY_AND_DISK\",\n    ) -> DataFrame:\n        \"\"\"Cache table with specified storage level.\"\"\"\n        if table_name in self._cache_registry:\n            return self._cache_registry[table_name]\n\n        df = self.spark.table(table_name)\n\n        if cache_level == \"MEMORY_ONLY\":\n            df.cache()\n        elif cache_level == \"MEMORY_AND_DISK\":\n            from pyspark import StorageLevel\n            df.persist(StorageLevel.MEMORY_AND_DISK)\n        elif cache_level == \"DISK_ONLY\":\n            from pyspark import StorageLevel\n            df.persist(StorageLevel.DISK_ONLY)\n\n        # Trigger caching\n        df.count()\n\n        self._cache_registry[table_name] = df\n        return df\n\n    def uncache_all(self):\n        \"\"\"Clear all cached DataFrames.\"\"\"\n        for df in self._cache_registry.values():\n            df.unpersist()\n        self._cache_registry.clear()\n        self.spark.catalog.clearCache()\n\n# Delta Cache (automatic)\n# Enable in cluster config:\n# \"spark.databricks.io.cache.enabled\": \"true\"\n# \"spark.databricks.io.cache.maxDiskUsage\": \"50g\"\n```\n\n### Step 6: Join Optimization\n\n```python\nfrom pyspark.sql import DataFrame\nfrom pyspark.sql.functions import broadcast\n\ndef optimize_join(\n    df_large: DataFrame,\n    df_small: DataFrame,\n    join_key: str,\n    small_table_threshold_mb: int = 100,\n) -> DataFrame:\n    \"\"\"\n    Optimize join based on table sizes.\n\n    Uses broadcast join for small tables,\n    sort-merge join for large tables.\n    \"\"\"\n    # Estimate small table size\n    small_size_mb = df_small.count() * 100 \u002F 1024 \u002F 1024  # rough estimate\n\n    if small_size_mb \u003C small_table_threshold_mb:\n        # Broadcast join (small table fits in memory)\n        return df_large.join(broadcast(df_small), join_key)\n    else:\n        # Sort-merge join with bucketing hint\n        return df_large.join(df_small, join_key, \"inner\")\n\n# Bucketed tables for frequent joins\ndef create_bucketed_table(\n    spark: SparkSession,\n    df: DataFrame,\n    table_name: str,\n    bucket_columns: list[str],\n    num_buckets: int = 100,\n):\n    \"\"\"Create bucketed table for join optimization.\"\"\"\n    (\n        df.write\n        .bucketBy(num_buckets, *bucket_columns)\n        .sortBy(*bucket_columns)\n        .saveAsTable(table_name)\n    )\n```\n\n## Output\n- Optimized cluster configuration\n- Tuned Spark settings\n- Optimized Delta tables\n- Improved query performance\n\n## Error Handling\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| OOM errors | Insufficient memory | Increase executor memory or reduce partition size |\n| Skewed data | Uneven distribution | Use salting or AQE skew handling |\n| Slow joins | Large shuffle | Use broadcast for small tables |\n| Too many files | Small files problem | Run OPTIMIZE regularly |\n\n## Examples\n\n### Performance Benchmark\n```python\nimport time\n\ndef benchmark_query(spark, query: str, runs: int = 3) -> dict:\n    \"\"\"Benchmark query execution time.\"\"\"\n    times = []\n    for _ in range(runs):\n        spark.catalog.clearCache()\n        start = time.time()\n        spark.sql(query).collect()\n        times.append(time.time() - start)\n\n    return {\n        \"min\": min(times),\n        \"max\": max(times),\n        \"avg\": sum(times) \u002F len(times),\n        \"runs\": runs,\n    }\n```\n\n## Resources\n- [Databricks Performance Guide](https:\u002F\u002Fdocs.databricks.com\u002Foptimizations\u002Findex.html)\n- [Delta Lake Optimization](https:\u002F\u002Fdocs.databricks.com\u002Fdelta\u002Foptimizations\u002Findex.html)\n- [Spark Tuning Guide](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Ftuning.html)\n\n## Next Steps\nFor cost optimization, see `databricks-cost-tuning`.\n",{"name":5,"author":46,"license":15,"version":47,"description":48,"allowed-tools":49},"Jeremy Longshore \u003Cjeremy@intentsolutions.io>","1.0.0","Optimize Databricks cluster and query performance.\nUse when jobs are running slowly, optimizing Spark configurations,\nor improving Delta Lake query performance.\nTrigger with phrases like \"databricks performance\", \"spark tuning\",\n\"databricks slow\", \"optimize databricks\", \"cluster performance\".\n","Read, Write, Edit, Bash(databricks:*), Grep","main","90b50ed0778745a80fd5bfb13b1843d45b9e82f7","tests\u002Fext_conformance\u002Fartifacts\u002Fplugins-community\u002Fplugins\u002Fsaas-packs\u002Fdatabricks-pack\u002Fskills\u002Fdatabricks-performance-tuning",231,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust","https:\u002F\u002Fgithub.com\u002FDicklesworthstone","https:\u002F\u002Fwww.jeffreyemanuel.com\u002F","Building in NY","2026-11-12 03:15:57",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add Dicklesworthstone\u002Fpi_agent_rust --skill databricks-performance-tuning",{"tool":65,"command":66},"bunx","bunx skills add Dicklesworthstone\u002Fpi_agent_rust --skill databricks-performance-tuning",{"tool":68,"command":69},"pnpm","pnpm dlx skills add Dicklesworthstone\u002Fpi_agent_rust --skill databricks-performance-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Fplugins-community\u002Fplugins\u002Fsaas-packs\u002Fdatabricks-pack\u002Fskills\u002Fdatabricks-performance-tuning\u002FSKILL.md",true,[],[74,114,138,164,191,212],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":79,"sourceHtmlUrl":80,"categories":81,"keywords":87,"publishedAt":41},"01M6FEGPPK0ZTKCCM0R3E3KS33","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FSKILL.md",[82],{"categoryNo":83,"categoryName":84,"taxonomyType":85,"taxonomyTypeDesc":86},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",[88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,33,106,107,108,109,110,111,112,113],"web artifacts builder","web","artifacts","artifacts builder","builder","frontend","css","components","react","component","routing","state management","state","shadcn","file","ui","Tools","Tailwind","html","single-file","simple","ui components","multi","for","jsx","suite",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":118,"sourceHtmlUrl":119,"categories":120,"keywords":125,"publishedAt":41},"01M6FEE9VTK1W4GY21AKREJXRG","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FSKILL.md",[121,124],{"categoryNo":122,"categoryName":123,"taxonomyType":85,"taxonomyTypeDesc":86},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":83,"categoryName":84,"taxonomyType":85,"taxonomyTypeDesc":86},[126,127,128,93,129,130,89,131,132,133,103,134,135,136,137,26,111],"testing","webapp testing","webapp","screenshots","playwright","debugging","toolkit","browser","logs","applications","behavior","local",{"skillNo":139,"name":140,"nameZh":6,"description":141,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":142,"sourceHtmlUrl":143,"categories":144,"keywords":145,"publishedAt":41},"01M6FE20F7ZTQBFWKF4MPJ2SBX","neuropixels-analysis","Neuropixels neural recording analysis. Load SpikeGLX\u002FOpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen\u002FIBL curation, AI-assisted visual analysis, for Neuropixels 1.0\u002F2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fscientific\u002Fneuropixels-analysis\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fscientific\u002Fneuropixels-analysis\u002FSKILL.md",[],[146,147,148,149,150,151,152,153,33,154,155,156,157,158,159,160,26,161,162,163,111,25],"analysis","neuropixels analysis","spike","metrics","data","open","quality","user","recordings","Neural","motion","visual","quality metrics","unit","sorting","curation","recording","load",{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":168,"sourceHtmlUrl":169,"categories":170,"keywords":174,"publishedAt":41},"01M6FDP0PDK6F1GGX395C7625Z","skill-creator","Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fskill-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fskill-creator\u002FSKILL.md",[171],{"categoryNo":172,"categoryName":173,"taxonomyType":85,"taxonomyTypeDesc":86},"01JBSEEDCAT000000000000023","Skill与智能体开发",[175,176,177,178,179,180,181,182,183,184,185,186,187,188,26,111,189,25,190],"skill creator","creator","workflows","create","update","capabilities","knowledge","tool","integrations","specialized","guide","new skill","existing skill","users","Existing","new",{"skillNo":192,"name":193,"nameZh":6,"description":194,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":195,"sourceHtmlUrl":196,"categories":197,"keywords":199,"publishedAt":41},"01M6FDM8Y7VK7NYG9SM7QPA3PW","notebooklm","Use this skill to query your Google NotebookLM notebooks directly from Claude Code for source-grounded, citation-backed answers from Gemini. Browser automation, library management, persistent auth. Drastically reduced hallucinations through document-only responses.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fnotebooklm\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fnotebooklm\u002FSKILL.md",[198],{"categoryNo":172,"categoryName":173,"taxonomyType":85,"taxonomyTypeDesc":86},[193,200,201,202,203,204,205,133,206,207,33,208,209,30,210,111,211],"google","browser automation","gemini","automation","auth","code","library","management","only","document","Citation","source",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":216,"sourceHtmlUrl":217,"categories":218,"keywords":225,"publishedAt":41},"01M6FDHPXE10J3XGTCH5HBHS1M","slack-gif-creator","Knowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like \"make me a GIF of X doing Y for Slack.\"","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fslack-gif-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fslack-gif-creator\u002FSKILL.md",[219,222],{"categoryNo":220,"categoryName":221,"taxonomyType":85,"taxonomyTypeDesc":86},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":223,"categoryName":224,"taxonomyType":85,"taxonomyTypeDesc":86},"01JBSEEDCAT000000000000016","图像与多媒体",[226,227,228,176,229,104,230,231,181,33,232,188,233,234,235,111,236,25],"slack gif creator","slack","gif","validation","constraints","Animation","me","make","Utilities","concepts","request"]