databricks-prod-checklist

Databricks Production Checklist

Overview

Complete checklist for deploying Databricks jobs and pipelines to production.

Prerequisites

• Staging environment tested and verified
• Production workspace access
• Unity Catalog configured
• Monitoring and alerting ready

Instructions

Step 1: Pre-Deployment Configuration

Security

• [ ] Service principal configured for automation
• [ ] Secrets in Databricks Secret Scopes (not env vars)
• [ ] Token expiration set (max 90 days)
• [ ] Unity Catalog permissions configured
• [ ] Cluster policies enforced
• [ ] IP access lists configured

Infrastructure

• [ ] Production cluster pool configured
• [ ] Instance types validated for workload
• [ ] Autoscaling configured appropriately
• [ ] Spot instance ratio set (cost vs reliability)

Step 2: Code Quality Verification

Testing

• [ ] Unit tests passing
• [ ] Integration tests on staging data
• [ ] Data quality tests defined
• [ ] Performance benchmarks met

# Run tests via Asset Bundles
databricks bundle validate -t prod
databricks bundle run -t staging test-job

# Verify test results
databricks runs get --run-id $RUN_ID | jq '.state.result_state'

Code Review

• [ ] No hardcoded credentials
• [ ] Error handling covers all failure modes
• [ ] Logging is production-appropriate
• [ ] Delta Lake best practices followed
• [ ] No collect() on large datasets

Step 3: Job Configuration

# resources/prod_job.yml
resources:
  jobs:
    etl_pipeline:
      name: "prod-etl-pipeline"
      tags:
        environment: production
        team: data-engineering
        cost_center: analytics

      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "America/New_York"

      email_notifications:
        on_failure:
          - "oncall@company.com"
        on_success:
          - "data-team@company.com"

      webhook_notifications:
        on_failure:
          - id: "slack-webhook-id"

      max_concurrent_runs: 1
      timeout_seconds: 14400  # 4 hours

      tasks:
        - task_key: bronze_ingest
          job_cluster_key: etl_cluster
          notebook_task:
            notebook_path: /Repos/prod/pipelines/bronze
          timeout_seconds: 3600

        - task_key: silver_transform
          depends_on:
            - task_key: bronze_ingest
          job_cluster_key: etl_cluster
          notebook_task:
            notebook_path: /Repos/prod/pipelines/silver

      job_clusters:
        - job_cluster_key: etl_cluster
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            node_type_id: "Standard_DS3_v2"
            num_workers: 4
            autoscale:
              min_workers: 2
              max_workers: 8
            spark_conf:
              spark.sql.shuffle.partitions: "200"
              spark.databricks.delta.optimizeWrite.enabled: "true"
            instance_pool_id: "prod-pool-id"

Step 4: Deployment Commands

# Pre-flight checks
echo "=== Pre-flight Checks ==="
databricks workspace list /Repos/prod/  # Verify repo exists
databricks clusters list | grep prod    # Verify pools/clusters
databricks secrets list-scopes          # Verify secrets

# Deploy with Asset Bundles
echo "=== Deploying ==="
databricks bundle deploy -t prod

# Verify deployment
databricks bundle summary -t prod
databricks jobs list | grep prod-etl

# Manual trigger to verify
echo "=== Verification Run ==="
RUN_ID=$(databricks jobs run-now --job-id $JOB_ID | jq -r '.run_id')
echo "Run ID: $RUN_ID"

# Monitor run
databricks runs get --run-id $RUN_ID --wait

Step 5: Monitoring Setup

# monitoring/health_check.py
from databricks.sdk import WorkspaceClient
from datetime import datetime, timedelta

def check_job_health(w: WorkspaceClient, job_id: int) -> dict:
    """Check job health metrics."""
    # Get recent runs
    runs = list(w.jobs.list_runs(
        job_id=job_id,
        completed_only=True,
        limit=10,
    ))

    if not runs:
        return {"status": "NO_RUNS", "healthy": False}

    # Calculate success rate
    successful = sum(1 for r in runs if r.state.result_state == "SUCCESS")
    success_rate = successful / len(runs)

    # Calculate average duration
    durations = [
        (r.end_time - r.start_time) / 1000 / 60  # minutes
        for r in runs if r.end_time
    ]
    avg_duration = sum(durations) / len(durations) if durations else 0

    # Check last run
    last_run = runs[0]
    last_state = last_run.state.result_state

    return {
        "status": "HEALTHY" if success_rate > 0.9 else "DEGRADED",
        "healthy": success_rate > 0.9 and last_state == "SUCCESS",
        "success_rate": success_rate,
        "avg_duration_minutes": avg_duration,
        "last_run_state": last_state,
        "last_run_time": datetime.fromtimestamp(last_run.start_time / 1000),
    }

Step 6: Rollback Procedure

#!/bin/bash
# rollback.sh - Emergency rollback procedure

JOB_ID=$1
PREVIOUS_VERSION=$2

echo "=== ROLLBACK INITIATED ==="
echo "Job: $JOB_ID"
echo "Target Version: $PREVIOUS_VERSION"

# 1. Pause the job
echo "Pausing job..."
databricks jobs update --job-id $JOB_ID --json '{"settings": {"schedule": null}}'

# 2. Cancel active runs
echo "Cancelling active runs..."
databricks runs list --job-id $JOB_ID --active-only | \
  jq -r '.runs[].run_id' | \
  xargs -I {} databricks runs cancel --run-id {}

# 3. Reset to previous version
echo "Rolling back to version $PREVIOUS_VERSION..."
databricks bundle deploy -t prod --force

# 4. Re-enable schedule
echo "Re-enabling schedule..."
# (restore from backup config)

# 5. Trigger verification run
echo "Triggering verification run..."
databricks jobs run-now --job-id $JOB_ID

echo "=== ROLLBACK COMPLETE ==="

Output

• Deployed production job
• Health checks passing
• Monitoring active
• Rollback procedure documented

Error Handling

Alert	Condition	Severity
Job Failed	result_state = FAILED	P1
Long Running	Duration > 2x average	P2
Consecutive Failures	3+ failures in a row	P1
Data Quality	Expectations failed	P2

Examples

Production Health Dashboard Query

-- Job health metrics (Unity Catalog system tables)
SELECT
  job_id,
  job_name,
  COUNT(*) as total_runs,
  SUM(CASE WHEN result_state = 'SUCCESS' THEN 1 ELSE 0 END) as successes,
  AVG(execution_duration) / 60000 as avg_minutes,
  MAX(start_time) as last_run
FROM system.lakeflow.job_run_timeline
WHERE start_time > current_timestamp() - INTERVAL 7 DAYS
GROUP BY job_id, job_name
ORDER BY total_runs DESC

Pre-Production Verification

# Comprehensive pre-prod check
databricks bundle validate -t prod && \
databricks bundle deploy -t prod --dry-run && \
echo "Validation passed, ready to deploy"

Resources

• Databricks Production Best Practices
• Job Configuration
• Monitoring Guide

Next Steps

For version upgrades, see databricks-upgrade-migration.