[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79AC5N4B8SVRW8VVAH3HPTP":3,"skill-same:01M79AC5N4B8SVRW8VVAH3HPTP":60},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":26,"commitSha":27,"skillDirectory":28,"forksCount":14,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M79AC5N4B8SVRW8VVAH3HPTP","ml-pipeline-automation",null,"Automate ML workflows with Airflow, Kubeflow, MLflow. Use for reproducible pipelines, retraining schedules, MLOps, or encountering task failures, dependency errors, experiment tracking issues.","Elogic360\u002Fsoftware-factory","Elogic360","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F156535519?v=4","01M74NH3F0RB13PWB7JZDE2V80","elogic360","01M74NH3FPC65082MDZVMFHDM3",0,"MIT","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fintegrations\u002Fclaude-skills-secondsky\u002Fplugins\u002Fml-pipeline-automation\u002Fskills\u002Fml-pipeline-automation\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: ml-pipeline-automation\ndescription: Automate ML workflows with Airflow, Kubeflow, MLflow. Use for reproducible pipelines, retraining schedules, MLOps, or encountering task failures, dependency errors, experiment tracking issues.\nlicense: MIT\nmetadata:\n  keywords: \"ML pipeline, Airflow, Kubeflow, MLflow, MLOps, workflow orchestration, data pipeline, model training automation, experiment tracking, model registry, Airflow DAG, task dependencies, pipeline monitoring, data quality, drift detection, hyperparameter tuning, model versioning, artifact management, Kubeflow Pipelines, pipeline automation, retries, sensors\"\n---\n\n# ML Pipeline Automation\n\nOrchestrate end-to-end machine learning workflows from data ingestion to production deployment with production-tested Airflow, Kubeflow, and MLflow patterns.\n\n## When to Use This Skill\n\nLoad this skill when:\n- **Building ML Pipelines**: Orchestrating data → train → deploy workflows\n- **Scheduling Retraining**: Setting up automated model retraining schedules\n- **Experiment Tracking**: Tracking experiments, parameters, metrics across runs\n- **MLOps Implementation**: Building reproducible, monitored ML infrastructure\n- **Workflow Orchestration**: Managing complex multi-step ML workflows\n- **Model Registry**: Managing model versions and deployment lifecycle\n\n## Quick Start: ML Pipeline in 5 Steps\n\n```bash\n# 1. Install Airflow and MLflow (check for latest versions at time of use)\npip install apache-airflow==3.1.5 mlflow==3.7.0\n\n# Note: These versions are current as of December 2025\n# Check PyPI for latest stable releases: https:\u002F\u002Fpypi.org\u002Fproject\u002Fapache-airflow\u002F\n\n# 2. Initialize Airflow database\nairflow db init\n\n# 3. Create DAG file: dags\u002Fml_training_pipeline.py\ncat > dags\u002Fml_training_pipeline.py \u003C\u003C 'EOF'\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom datetime import datetime, timedelta\n\ndefault_args = {\n    'owner': 'ml-team',\n    'retries': 2,\n    'retry_delay': timedelta(minutes=5)\n}\n\ndag = DAG(\n    'ml_training_pipeline',\n    default_args=default_args,\n    schedule_interval='@daily',\n    start_date=datetime(2025, 1, 1)\n)\n\ndef train_model(**context):\n    import mlflow\n    import mlflow.sklearn\n    from sklearn.ensemble import RandomForestClassifier\n    from sklearn.datasets import load_iris\n    from sklearn.model_selection import train_test_split\n\n    X, y = load_iris(return_X_y=True)\n    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n\n    mlflow.set_tracking_uri('http:\u002F\u002Flocalhost:5000')\n    mlflow.set_experiment('iris-training')\n\n    with mlflow.start_run():\n        model = RandomForestClassifier(n_estimators=100)\n        model.fit(X_train, y_train)\n\n        accuracy = model.score(X_test, y_test)\n        mlflow.log_metric('accuracy', accuracy)\n        mlflow.sklearn.log_model(model, 'model')\n\ntrain = PythonOperator(\n    task_id='train_model',\n    python_callable=train_model,\n    dag=dag\n)\nEOF\n\n# 4. Start Airflow scheduler and webserver\nairflow scheduler &\nairflow webserver --port 8080 &\n\n# 5. Trigger pipeline\nairflow dags trigger ml_training_pipeline\n\n# Access UI: http:\u002F\u002Flocalhost:8080\n```\n\n**Result**: Working ML pipeline with experiment tracking in under 5 minutes.\n\n## Core Concepts\n\n### Pipeline Stages\n\n1. **Data Collection** → Fetch raw data from sources\n2. **Data Validation** → Check schema, quality, distributions\n3. **Feature Engineering** → Transform raw data to features\n4. **Model Training** → Train with hyperparameter tuning\n5. **Model Evaluation** → Validate performance on test set\n6. **Model Deployment** → Push to production if metrics pass\n7. **Monitoring** → Track drift, performance in production\n\n### Orchestration Tools Comparison\n\n| Tool | Best For | Strengths |\n|------|----------|-----------|\n| **Airflow** | General ML workflows | Mature, flexible, Python-native |\n| **Kubeflow** | Kubernetes-native ML | Container-based, scalable |\n| **MLflow** | Experiment tracking | Model registry, versioning |\n| **Prefect** | Modern Python workflows | Dynamic DAGs, native caching |\n| **Dagster** | Asset-oriented pipelines | Data-aware, testable |\n\n## Basic Airflow DAG\n\n```python\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom datetime import datetime, timedelta\nimport logging\n\nlogger = logging.getLogger(__name__)\n\ndefault_args = {\n    'owner': 'ml-team',\n    'depends_on_past': False,\n    'email': ['alerts@example.com'],\n    'email_on_failure': True,\n    'retries': 2,\n    'retry_delay': timedelta(minutes=5)\n}\n\ndag = DAG(\n    'ml_training_pipeline',\n    default_args=default_args,\n    description='End-to-end ML training pipeline',\n    schedule_interval='@daily',\n    start_date=datetime(2025, 1, 1),\n    catchup=False\n)\n\ndef validate_data(**context):\n    \"\"\"Validate input data quality.\"\"\"\n    import pandas as pd\n\n    data_path = \"\u002Fdata\u002Fraw\u002Flatest.csv\"\n    df = pd.read_csv(data_path)\n\n    # Validation checks\n    assert len(df) > 1000, f\"Insufficient data: {len(df)} rows\"\n    assert df.isnull().sum().sum() \u003C len(df) * 0.1, \"Too many nulls\"\n\n    context['ti'].xcom_push(key='data_path', value=data_path)\n    logger.info(f\"Data validation passed: {len(df)} rows\")\n\ndef train_model(**context):\n    \"\"\"Train ML model with MLflow tracking.\"\"\"\n    import mlflow\n    import mlflow.sklearn\n    from sklearn.ensemble import RandomForestClassifier\n\n    data_path = context['ti'].xcom_pull(key='data_path', task_ids='validate_data')\n\n    mlflow.set_tracking_uri('http:\u002F\u002Fmlflow:5000')\n    mlflow.set_experiment('production-training')\n\n    with mlflow.start_run():\n        # Training logic here\n        model = RandomForestClassifier(n_estimators=100)\n        # model.fit(X, y) ...\n\n        mlflow.log_param('n_estimators', 100)\n        mlflow.sklearn.log_model(model, 'model')\n\nvalidate = PythonOperator(\n    task_id='validate_data',\n    python_callable=validate_data,\n    dag=dag\n)\n\ntrain = PythonOperator(\n    task_id='train_model',\n    python_callable=train_model,\n    dag=dag\n)\n\nvalidate >> train\n```\n\n## Known Issues Prevention\n\n### 1. Task Failures Without Alerts\n**Problem**: Pipeline fails silently, no one notices until users complain.\n\n**Solution**: Configure email\u002FSlack alerts on failure:\n```python\ndefault_args = {\n    'email': ['ml-team@example.com'],\n    'email_on_failure': True,\n    'email_on_retry': False\n}\n\ndef on_failure_callback(context):\n    \"\"\"Send Slack alert on failure.\"\"\"\n    from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator\n\n    slack_msg = f\"\"\"\n    :red_circle: Task Failed: {context['task_instance'].task_id}\n    DAG: {context['task_instance'].dag_id}\n    Execution Date: {context['ds']}\n    Error: {context.get('exception')}\n    \"\"\"\n\n    SlackWebhookOperator(\n        task_id='slack_alert',\n        slack_webhook_conn_id='slack_webhook',\n        message=slack_msg\n    ).execute(context)\n\ntask = PythonOperator(\n    task_id='critical_task',\n    python_callable=my_function,\n    on_failure_callback=on_failure_callback,\n    dag=dag\n)\n```\n\n### 2. Missing XCom Data Between Tasks\n**Problem**: Task expects XCom value from previous task, gets None, crashes.\n\n**Solution**: Always validate XCom pulls:\n```python\ndef process_data(**context):\n    data_path = context['ti'].xcom_pull(\n        key='data_path',\n        task_ids='upstream_task'\n    )\n\n    if data_path is None:\n        raise ValueError(\"No data_path from upstream_task - check XCom push\")\n\n    # Process data...\n```\n\n### 3. DAG Not Appearing in UI\n**Problem**: DAG file exists in `dags\u002F` but doesn't show in Airflow UI.\n\n**Solution**: Check DAG parsing errors:\n```bash\n# Check for syntax errors\npython dags\u002Fmy_dag.py\n\n# View DAG import errors in UI\n# Navigate to: Browse → DAG Import Errors\n\n# Common fixes:\n# 1. Ensure DAG object is defined in file\n# 2. Check for circular imports\n# 3. Verify all dependencies installed\n# 4. Fix syntax errors\n```\n\n### 4. Hardcoded Paths Break in Production\n**Problem**: Paths like `\u002FUsers\u002Fmyname\u002Fdata\u002F` work locally, fail in production.\n\n**Solution**: Use Airflow Variables or environment variables:\n```python\nfrom airflow.models import Variable\n\ndef load_data(**context):\n    # ❌ Bad: Hardcoded path\n    # data_path = \"\u002FUsers\u002Fmyname\u002Fdata\u002Ftrain.csv\"\n\n    # ✅ Good: Use Airflow Variable\n    data_dir = Variable.get(\"data_directory\", \"\u002Fdata\")\n    data_path = f\"{data_dir}\u002Ftrain.csv\"\n\n    # Or use environment variable\n    import os\n    data_path = os.getenv(\"DATA_PATH\", \"\u002Fdata\u002Ftrain.csv\")\n```\n\n### 5. Stuck Tasks Consume Resources\n**Problem**: Task hangs indefinitely, blocks worker slot, wastes resources.\n\n**Solution**: Set execution_timeout on tasks:\n```python\nfrom datetime import timedelta\n\ntask = PythonOperator(\n    task_id='long_running_task',\n    python_callable=my_function,\n    execution_timeout=timedelta(hours=2),  # Kill after 2 hours\n    dag=dag\n)\n```\n\n### 6. No Data Validation = Bad Model Training\n**Problem**: Train on corrupted\u002Fincomplete data, model performs poorly in production.\n\n**Solution**: Add data quality validation tasks:\n```python\ndef validate_data_quality(**context):\n    \"\"\"Comprehensive data validation.\"\"\"\n    import pandas as pd\n\n    df = pd.read_csv(data_path)\n\n    # Schema validation\n    required_cols = ['user_id', 'timestamp', 'feature_a', 'target']\n    missing_cols = set(required_cols) - set(df.columns)\n    if missing_cols:\n        raise ValueError(f\"Missing columns: {missing_cols}\")\n\n    # Statistical validation\n    if df['target'].isnull().sum() > 0:\n        raise ValueError(\"Target column contains nulls\")\n\n    if len(df) \u003C 1000:\n        raise ValueError(f\"Insufficient data: {len(df)} rows\")\n\n    logger.info(\"✅ Data quality validation passed\")\n```\n\n### 7. Untracked Experiments = Lost Knowledge\n**Problem**: Can't reproduce results, don't know which hyperparameters worked.\n\n**Solution**: Use MLflow for all experiments:\n```python\nimport mlflow\n\nmlflow.set_tracking_uri('http:\u002F\u002Fmlflow:5000')\nmlflow.set_experiment('model-experiments')\n\nwith mlflow.start_run(run_name='rf_v1'):\n    # Log ALL hyperparameters\n    mlflow.log_params({\n        'model_type': 'random_forest',\n        'n_estimators': 100,\n        'max_depth': 10,\n        'random_state': 42\n    })\n\n    # Log ALL metrics\n    mlflow.log_metrics({\n        'train_accuracy': 0.95,\n        'test_accuracy': 0.87,\n        'f1_score': 0.89\n    })\n\n    # Log model\n    mlflow.sklearn.log_model(model, 'model')\n```\n\n## When to Load References\n\nLoad reference files for detailed production implementations:\n\n- **Airflow DAG Patterns**: Load `references\u002Fairflow-patterns.md` when building complex DAGs with error handling, dynamic generation, sensors, task groups, or retry logic. Contains complete production DAG examples.\n\n- **Kubeflow & MLflow Integration**: Load `references\u002Fkubeflow-mlflow.md` when using Kubeflow Pipelines for container-native orchestration, integrating MLflow tracking, building KFP components, or managing model registry.\n\n- **Pipeline Monitoring**: Load `references\u002Fpipeline-monitoring.md` when implementing data quality checks, drift detection, alert configuration, or pipeline health monitoring with Prometheus.\n\n## Best Practices\n\n1. **Idempotent Tasks**: Tasks should produce same result when re-run\n2. **Atomic Operations**: Each task does one thing well\n3. **Version Everything**: Data, code, models, dependencies\n4. **Comprehensive Logging**: Log all important events with context\n5. **Error Handling**: Fail fast with clear error messages\n6. **Monitoring**: Track pipeline health, data quality, model drift\n7. **Testing**: Test tasks independently before integrating\n8. **Documentation**: Document DAG purpose, task dependencies\n\n## Common Patterns\n\n### Conditional Execution\n```python\nfrom airflow.operators.python import BranchPythonOperator\n\ndef choose_branch(**context):\n    accuracy = context['ti'].xcom_pull(key='accuracy', task_ids='evaluate')\n\n    if accuracy > 0.9:\n        return 'deploy_to_production'\n    else:\n        return 'retrain_with_more_data'\n\nbranch = BranchPythonOperator(\n    task_id='check_accuracy',\n    python_callable=choose_branch,\n    dag=dag\n)\n\ntrain >> evaluate >> branch >> [deploy, retrain]\n```\n\n### Parallel Training\n```python\nfrom airflow.utils.task_group import TaskGroup\n\nwith TaskGroup('train_models', dag=dag) as train_group:\n    train_rf = PythonOperator(task_id='train_rf', ...)\n    train_lr = PythonOperator(task_id='train_lr', ...)\n    train_xgb = PythonOperator(task_id='train_xgb', ...)\n\n# All models train in parallel\npreprocess >> train_group >> select_best\n```\n\n### Waiting for Data\n```python\nfrom airflow.sensors.filesystem import FileSensor\n\nwait_for_data = FileSensor(\n    task_id='wait_for_data',\n    filepath='\u002Fdata\u002Finput\u002F{{ ds }}.csv',\n    poke_interval=60,  # Check every 60 seconds\n    timeout=3600,  # Timeout after 1 hour\n    mode='reschedule',  # Don't block worker\n    dag=dag\n)\n\nwait_for_data >> process_data\n```\n","# ML Pipeline Automation\n\nOrchestrate end-to-end machine learning workflows from data ingestion to production deployment with production-tested Airflow, Kubeflow, and MLflow patterns.\n\n## When to Use This Skill\n\nLoad this skill when:\n- **Building ML Pipelines**: Orchestrating data → train → deploy workflows\n- **Scheduling Retraining**: Setting up automated model retraining schedules\n- **Experiment Tracking**: Tracking experiments, parameters, metrics across runs\n- **MLOps Implementation**: Building reproducible, monitored ML infrastructure\n- **Workflow Orchestration**: Managing complex multi-step ML workflows\n- **Model Registry**: Managing model versions and deployment lifecycle\n\n## Quick Start: ML Pipeline in 5 Steps\n\n```bash\n# 1. Install Airflow and MLflow (check for latest versions at time of use)\npip install apache-airflow==3.1.5 mlflow==3.7.0\n\n# Note: These versions are current as of December 2025\n# Check PyPI for latest stable releases: https:\u002F\u002Fpypi.org\u002Fproject\u002Fapache-airflow\u002F\n\n# 2. Initialize Airflow database\nairflow db init\n\n# 3. Create DAG file: dags\u002Fml_training_pipeline.py\ncat > dags\u002Fml_training_pipeline.py \u003C\u003C 'EOF'\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom datetime import datetime, timedelta\n\ndefault_args = {\n    'owner': 'ml-team',\n    'retries': 2,\n    'retry_delay': timedelta(minutes=5)\n}\n\ndag = DAG(\n    'ml_training_pipeline',\n    default_args=default_args,\n    schedule_interval='@daily',\n    start_date=datetime(2025, 1, 1)\n)\n\ndef train_model(**context):\n    import mlflow\n    import mlflow.sklearn\n    from sklearn.ensemble import RandomForestClassifier\n    from sklearn.datasets import load_iris\n    from sklearn.model_selection import train_test_split\n\n    X, y = load_iris(return_X_y=True)\n    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n\n    mlflow.set_tracking_uri('http:\u002F\u002Flocalhost:5000')\n    mlflow.set_experiment('iris-training')\n\n    with mlflow.start_run():\n        model = RandomForestClassifier(n_estimators=100)\n        model.fit(X_train, y_train)\n\n        accuracy = model.score(X_test, y_test)\n        mlflow.log_metric('accuracy', accuracy)\n        mlflow.sklearn.log_model(model, 'model')\n\ntrain = PythonOperator(\n    task_id='train_model',\n    python_callable=train_model,\n    dag=dag\n)\nEOF\n\n# 4. Start Airflow scheduler and webserver\nairflow scheduler &\nairflow webserver --port 8080 &\n\n# 5. Trigger pipeline\nairflow dags trigger ml_training_pipeline\n\n# Access UI: http:\u002F\u002Flocalhost:8080\n```\n\n**Result**: Working ML pipeline with experiment tracking in under 5 minutes.\n\n## Core Concepts\n\n### Pipeline Stages\n\n1. **Data Collection** → Fetch raw data from sources\n2. **Data Validation** → Check schema, quality, distributions\n3. **Feature Engineering** → Transform raw data to features\n4. **Model Training** → Train with hyperparameter tuning\n5. **Model Evaluation** → Validate performance on test set\n6. **Model Deployment** → Push to production if metrics pass\n7. **Monitoring** → Track drift, performance in production\n\n### Orchestration Tools Comparison\n\n| Tool | Best For | Strengths |\n|------|----------|-----------|\n| **Airflow** | General ML workflows | Mature, flexible, Python-native |\n| **Kubeflow** | Kubernetes-native ML | Container-based, scalable |\n| **MLflow** | Experiment tracking | Model registry, versioning |\n| **Prefect** | Modern Python workflows | Dynamic DAGs, native caching |\n| **Dagster** | Asset-oriented pipelines | Data-aware, testable |\n\n## Basic Airflow DAG\n\n```python\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom datetime import datetime, timedelta\nimport logging\n\nlogger = logging.getLogger(__name__)\n\ndefault_args = {\n    'owner': 'ml-team',\n    'depends_on_past': False,\n    'email': ['alerts@example.com'],\n    'email_on_failure': True,\n    'retries': 2,\n    'retry_delay': timedelta(minutes=5)\n}\n\ndag = DAG(\n    'ml_training_pipeline',\n    default_args=default_args,\n    description='End-to-end ML training pipeline',\n    schedule_interval='@daily',\n    start_date=datetime(2025, 1, 1),\n    catchup=False\n)\n\ndef validate_data(**context):\n    \"\"\"Validate input data quality.\"\"\"\n    import pandas as pd\n\n    data_path = \"\u002Fdata\u002Fraw\u002Flatest.csv\"\n    df = pd.read_csv(data_path)\n\n    # Validation checks\n    assert len(df) > 1000, f\"Insufficient data: {len(df)} rows\"\n    assert df.isnull().sum().sum() \u003C len(df) * 0.1, \"Too many nulls\"\n\n    context['ti'].xcom_push(key='data_path', value=data_path)\n    logger.info(f\"Data validation passed: {len(df)} rows\")\n\ndef train_model(**context):\n    \"\"\"Train ML model with MLflow tracking.\"\"\"\n    import mlflow\n    import mlflow.sklearn\n    from sklearn.ensemble import RandomForestClassifier\n\n    data_path = context['ti'].xcom_pull(key='data_path', task_ids='validate_data')\n\n    mlflow.set_tracking_uri('http:\u002F\u002Fmlflow:5000')\n    mlflow.set_experiment('production-training')\n\n    with mlflow.start_run():\n        # Training logic here\n        model = RandomForestClassifier(n_estimators=100)\n        # model.fit(X, y) ...\n\n        mlflow.log_param('n_estimators', 100)\n        mlflow.sklearn.log_model(model, 'model')\n\nvalidate = PythonOperator(\n    task_id='validate_data',\n    python_callable=validate_data,\n    dag=dag\n)\n\ntrain = PythonOperator(\n    task_id='train_model',\n    python_callable=train_model,\n    dag=dag\n)\n\nvalidate >> train\n```\n\n## Known Issues Prevention\n\n### 1. Task Failures Without Alerts\n**Problem**: Pipeline fails silently, no one notices until users complain.\n\n**Solution**: Configure email\u002FSlack alerts on failure:\n```python\ndefault_args = {\n    'email': ['ml-team@example.com'],\n    'email_on_failure': True,\n    'email_on_retry': False\n}\n\ndef on_failure_callback(context):\n    \"\"\"Send Slack alert on failure.\"\"\"\n    from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator\n\n    slack_msg = f\"\"\"\n    :red_circle: Task Failed: {context['task_instance'].task_id}\n    DAG: {context['task_instance'].dag_id}\n    Execution Date: {context['ds']}\n    Error: {context.get('exception')}\n    \"\"\"\n\n    SlackWebhookOperator(\n        task_id='slack_alert',\n        slack_webhook_conn_id='slack_webhook',\n        message=slack_msg\n    ).execute(context)\n\ntask = PythonOperator(\n    task_id='critical_task',\n    python_callable=my_function,\n    on_failure_callback=on_failure_callback,\n    dag=dag\n)\n```\n\n### 2. Missing XCom Data Between Tasks\n**Problem**: Task expects XCom value from previous task, gets None, crashes.\n\n**Solution**: Always validate XCom pulls:\n```python\ndef process_data(**context):\n    data_path = context['ti'].xcom_pull(\n        key='data_path',\n        task_ids='upstream_task'\n    )\n\n    if data_path is None:\n        raise ValueError(\"No data_path from upstream_task - check XCom push\")\n\n    # Process data...\n```\n\n### 3. DAG Not Appearing in UI\n**Problem**: DAG file exists in `dags\u002F` but doesn't show in Airflow UI.\n\n**Solution**: Check DAG parsing errors:\n```bash\n# Check for syntax errors\npython dags\u002Fmy_dag.py\n\n# View DAG import errors in UI\n# Navigate to: Browse → DAG Import Errors\n\n# Common fixes:\n# 1. Ensure DAG object is defined in file\n# 2. Check for circular imports\n# 3. Verify all dependencies installed\n# 4. Fix syntax errors\n```\n\n### 4. Hardcoded Paths Break in Production\n**Problem**: Paths like `\u002FUsers\u002Fmyname\u002Fdata\u002F` work locally, fail in production.\n\n**Solution**: Use Airflow Variables or environment variables:\n```python\nfrom airflow.models import Variable\n\ndef load_data(**context):\n    # ❌ Bad: Hardcoded path\n    # data_path = \"\u002FUsers\u002Fmyname\u002Fdata\u002Ftrain.csv\"\n\n    # ✅ Good: Use Airflow Variable\n    data_dir = Variable.get(\"data_directory\", \"\u002Fdata\")\n    data_path = f\"{data_dir}\u002Ftrain.csv\"\n\n    # Or use environment variable\n    import os\n    data_path = os.getenv(\"DATA_PATH\", \"\u002Fdata\u002Ftrain.csv\")\n```\n\n### 5. Stuck Tasks Consume Resources\n**Problem**: Task hangs indefinitely, blocks worker slot, wastes resources.\n\n**Solution**: Set execution_timeout on tasks:\n```python\nfrom datetime import timedelta\n\ntask = PythonOperator(\n    task_id='long_running_task',\n    python_callable=my_function,\n    execution_timeout=timedelta(hours=2),  # Kill after 2 hours\n    dag=dag\n)\n```\n\n### 6. No Data Validation = Bad Model Training\n**Problem**: Train on corrupted\u002Fincomplete data, model performs poorly in production.\n\n**Solution**: Add data quality validation tasks:\n```python\ndef validate_data_quality(**context):\n    \"\"\"Comprehensive data validation.\"\"\"\n    import pandas as pd\n\n    df = pd.read_csv(data_path)\n\n    # Schema validation\n    required_cols = ['user_id', 'timestamp', 'feature_a', 'target']\n    missing_cols = set(required_cols) - set(df.columns)\n    if missing_cols:\n        raise ValueError(f\"Missing columns: {missing_cols}\")\n\n    # Statistical validation\n    if df['target'].isnull().sum() > 0:\n        raise ValueError(\"Target column contains nulls\")\n\n    if len(df) \u003C 1000:\n        raise ValueError(f\"Insufficient data: {len(df)} rows\")\n\n    logger.info(\"✅ Data quality validation passed\")\n```\n\n### 7. Untracked Experiments = Lost Knowledge\n**Problem**: Can't reproduce results, don't know which hyperparameters worked.\n\n**Solution**: Use MLflow for all experiments:\n```python\nimport mlflow\n\nmlflow.set_tracking_uri('http:\u002F\u002Fmlflow:5000')\nmlflow.set_experiment('model-experiments')\n\nwith mlflow.start_run(run_name='rf_v1'):\n    # Log ALL hyperparameters\n    mlflow.log_params({\n        'model_type': 'random_forest',\n        'n_estimators': 100,\n        'max_depth': 10,\n        'random_state': 42\n    })\n\n    # Log ALL metrics\n    mlflow.log_metrics({\n        'train_accuracy': 0.95,\n        'test_accuracy': 0.87,\n        'f1_score': 0.89\n    })\n\n    # Log model\n    mlflow.sklearn.log_model(model, 'model')\n```\n\n## When to Load References\n\nLoad reference files for detailed production implementations:\n\n- **Airflow DAG Patterns**: Load `references\u002Fairflow-patterns.md` when building complex DAGs with error handling, dynamic generation, sensors, task groups, or retry logic. Contains complete production DAG examples.\n\n- **Kubeflow & MLflow Integration**: Load `references\u002Fkubeflow-mlflow.md` when using Kubeflow Pipelines for container-native orchestration, integrating MLflow tracking, building KFP components, or managing model registry.\n\n- **Pipeline Monitoring**: Load `references\u002Fpipeline-monitoring.md` when implementing data quality checks, drift detection, alert configuration, or pipeline health monitoring with Prometheus.\n\n## Best Practices\n\n1. **Idempotent Tasks**: Tasks should produce same result when re-run\n2. **Atomic Operations**: Each task does one thing well\n3. **Version Everything**: Data, code, models, dependencies\n4. **Comprehensive Logging**: Log all important events with context\n5. **Error Handling**: Fail fast with clear error messages\n6. **Monitoring**: Track pipeline health, data quality, model drift\n7. **Testing**: Test tasks independently before integrating\n8. **Documentation**: Document DAG purpose, task dependencies\n\n## Common Patterns\n\n### Conditional Execution\n```python\nfrom airflow.operators.python import BranchPythonOperator\n\ndef choose_branch(**context):\n    accuracy = context['ti'].xcom_pull(key='accuracy', task_ids='evaluate')\n\n    if accuracy > 0.9:\n        return 'deploy_to_production'\n    else:\n        return 'retrain_with_more_data'\n\nbranch = BranchPythonOperator(\n    task_id='check_accuracy',\n    python_callable=choose_branch,\n    dag=dag\n)\n\ntrain >> evaluate >> branch >> [deploy, retrain]\n```\n\n### Parallel Training\n```python\nfrom airflow.utils.task_group import TaskGroup\n\nwith TaskGroup('train_models', dag=dag) as train_group:\n    train_rf = PythonOperator(task_id='train_rf', ...)\n    train_lr = PythonOperator(task_id='train_lr', ...)\n    train_xgb = PythonOperator(task_id='train_xgb', ...)\n\n# All models train in parallel\npreprocess >> train_group >> select_best\n```\n\n### Waiting for Data\n```python\nfrom airflow.sensors.filesystem import FileSensor\n\nwait_for_data = FileSensor(\n    task_id='wait_for_data',\n    filepath='\u002Fdata\u002Finput\u002F{{ ds }}.csv',\n    poke_interval=60,  # Check every 60 seconds\n    timeout=3600,  # Timeout after 1 hour\n    mode='reschedule',  # Don't block worker\n    dag=dag\n)\n\nwait_for_data >> process_data\n```\n",{"name":5,"license":15,"metadata":24,"description":7},{"keywords":25},"ML pipeline, Airflow, Kubeflow, MLflow, MLOps, workflow orchestration, data pipeline, model training automation, experiment tracking, model registry, Airflow DAG, task dependencies, pipeline monitoring, data quality, drift detection, hyperparameter tuning, model versioning, artifact management, Kubeflow Pipelines, pipeline automation, retries, sensors","main","c475fc64e94e803b50d14a791dfee8853fbb2c5c","integrations\u002Fclaude-skills-secondsky\u002Fplugins\u002Fml-pipeline-automation\u002Fskills\u002Fml-pipeline-automation","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory","https:\u002F\u002Fgithub.com\u002FElogic360","2026-11-08 22:21:22",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add Elogic360\u002Fsoftware-factory --skill ml-pipeline-automation",{"tool":38,"command":39},"bunx","bunx skills add Elogic360\u002Fsoftware-factory --skill ml-pipeline-automation",{"tool":41,"command":42},"pnpm","pnpm dlx skills add Elogic360\u002Fsoftware-factory --skill ml-pipeline-automation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fintegrations\u002Fclaude-skills-secondsky\u002Fplugins\u002Fml-pipeline-automation\u002Fskills\u002Fml-pipeline-automation\u002FSKILL.md",true,[46,52,56],{"path":47,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":50,"sourceHtmlUrl":51},"references\u002Fairflow-patterns.md","REFERENCE","参考文档",10970,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fintegrations\u002Fclaude-skills-secondsky\u002Fplugins\u002Fml-pipeline-automation\u002Fskills\u002Fml-pipeline-automation\u002Freferences\u002Fairflow-patterns.md",{"path":53,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002Fkubeflow-mlflow.md",15477,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fintegrations\u002Fclaude-skills-secondsky\u002Fplugins\u002Fml-pipeline-automation\u002Fskills\u002Fml-pipeline-automation\u002Freferences\u002Fkubeflow-mlflow.md",{"path":57,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fpipeline-monitoring.md",19213,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fintegrations\u002Fclaude-skills-secondsky\u002Fplugins\u002Fml-pipeline-automation\u002Fskills\u002Fml-pipeline-automation\u002Freferences\u002Fpipeline-monitoring.md",[61,69,76,83,90,97],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M79G206MT3J6S0GY3C45P2KV","vss-summarize-video","Use to summarize a recorded video via the LVS summarization microservice (HITL-gated) with a VLM fallback. Not for report generation or live RTSP captioning.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-summarize-video\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M79G1WN63PGRE8CKD47WEBQ5","vss-setup-video-analytics-api","Use to deploy the vss-video-analytics-api REST service standalone (config-source, data-log bind, Elasticsearch, optional Kafka). Not for full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-video-analytics-api\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M79G1S9F7VY3EF6H53PRP5AN","vss-setup-behavior-analytics","Use to deploy the vss-behavior-analytics service standalone (entrypoint, config-source, optional calibration). Not for the full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-behavior-analytics\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M79G1N1S1Y27PFZPYMNK1MTQ","vss-search-archive","Use this skill to run top-level VSS fusion search on archived video, or to ingest video files \u002F RTSP streams for search. Do NOT use for ad-hoc visual Q&A (use vss-ask-video), live captioning (use vss-deploy-dense-captioning), or video summarization and reports (use vss-summarize-video).","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-search-archive\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M79G1H6WN0Q6C0VCAXCZX6B7","vss-query-analytics","Use this skill when reading video-analytics metrics, incidents, alerts, and sensor data via the VA-MCP server (port 9901). Not for live VLM or incident-range narrative reports.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-query-analytics\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M79G1DYPNXQS7QHZ0AAF5FX5","vss-manage-video-io-storage","Use to call the VIOS REST API (sensor list, timelines, clip extraction, snapshots, add\u002Fdelete sensors and streams). Not for VLM inference or search.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-manage-video-io-storage\u002FSKILL.md",[],[]]