[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BWASWD83S1278MZP5TZSWK":3,"skill-same:01M7BWASWD83S1278MZP5TZSWK":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7BWASWD83S1278MZP5TZSWK","airflow-dag-patterns",null,"Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Plugin-插件协议\u002FYYC3-Plugin-插件协议-plugins-claude\u002Fdata-engineering\u002Fskills\u002Fairflow-dag-patterns\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: airflow-dag-patterns\ndescription: Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.\n---\n\n# Apache Airflow DAG Patterns\n\nProduction-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.\n\n## When to Use This Skill\n\n- Creating data pipeline orchestration with Airflow\n- Designing DAG structures and dependencies\n- Implementing custom operators and sensors\n- Testing Airflow DAGs locally\n- Setting up Airflow in production\n- Debugging failed DAG runs\n\n## Core Concepts\n\n### 1. DAG Design Principles\n\n| Principle       | Description                         |\n| --------------- | ----------------------------------- |\n| **Idempotent**  | Running twice produces same result  |\n| **Atomic**      | Tasks succeed or fail completely    |\n| **Incremental** | Process only new\u002Fchanged data       |\n| **Observable**  | Logs, metrics, alerts at every step |\n\n### 2. Task Dependencies\n\n```python\n# Linear\ntask1 >> task2 >> task3\n\n# Fan-out\ntask1 >> [task2, task3, task4]\n\n# Fan-in\n[task1, task2, task3] >> task4\n\n# Complex\ntask1 >> task2 >> task4\ntask1 >> task3 >> task4\n```\n\n## Quick Start\n\n```python\n# dags\u002Fexample_dag.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.operators.empty import EmptyOperator\n\ndefault_args = {\n    'owner': 'data-team',\n    'depends_on_past': False,\n    'email_on_failure': True,\n    'email_on_retry': False,\n    'retries': 3,\n    'retry_delay': timedelta(minutes=5),\n    'retry_exponential_backoff': True,\n    'max_retry_delay': timedelta(hours=1),\n}\n\nwith DAG(\n    dag_id='example_etl',\n    default_args=default_args,\n    description='Example ETL pipeline',\n    schedule='0 6 * * *',  # Daily at 6 AM\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'example'],\n    max_active_runs=1,\n) as dag:\n\n    start = EmptyOperator(task_id='start')\n\n    def extract_data(**context):\n        execution_date = context['ds']\n        # Extract logic here\n        return {'records': 1000}\n\n    extract = PythonOperator(\n        task_id='extract',\n        python_callable=extract_data,\n    )\n\n    end = EmptyOperator(task_id='end')\n\n    start >> extract >> end\n```\n\n## Patterns\n\n### Pattern 1: TaskFlow API (Airflow 2.0+)\n\n```python\n# dags\u002Ftaskflow_example.py\nfrom datetime import datetime\nfrom airflow.decorators import dag, task\nfrom airflow.models import Variable\n\n@dag(\n    dag_id='taskflow_etl',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'taskflow'],\n)\ndef taskflow_etl():\n    \"\"\"ETL pipeline using TaskFlow API\"\"\"\n\n    @task()\n    def extract(source: str) -> dict:\n        \"\"\"Extract data from source\"\"\"\n        import pandas as pd\n\n        df = pd.read_csv(f's3:\u002F\u002Fbucket\u002F{source}\u002F{{ ds }}.csv')\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def transform(extracted: dict) -> dict:\n        \"\"\"Transform extracted data\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(extracted['data'])\n        df['processed_at'] = datetime.now()\n        df = df.dropna()\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def load(transformed: dict, target: str):\n        \"\"\"Load data to target\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(transformed['data'])\n        df.to_parquet(f's3:\u002F\u002Fbucket\u002F{target}\u002F{{ ds }}.parquet')\n        return transformed['rows']\n\n    @task()\n    def notify(rows_loaded: int):\n        \"\"\"Send notification\"\"\"\n        print(f'Loaded {rows_loaded} rows')\n\n    # Define dependencies with XCom passing\n    extracted = extract(source='raw_data')\n    transformed = transform(extracted)\n    loaded = load(transformed, target='processed_data')\n    notify(loaded)\n\n# Instantiate the DAG\ntaskflow_etl()\n```\n\n### Pattern 2: Dynamic DAG Generation\n\n```python\n# dags\u002Fdynamic_dag_factory.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.models import Variable\nimport json\n\n# Configuration for multiple similar pipelines\nPIPELINE_CONFIGS = [\n    {'name': 'customers', 'schedule': '@daily', 'source': 's3:\u002F\u002Fraw\u002Fcustomers'},\n    {'name': 'orders', 'schedule': '@hourly', 'source': 's3:\u002F\u002Fraw\u002Forders'},\n    {'name': 'products', 'schedule': '@weekly', 'source': 's3:\u002F\u002Fraw\u002Fproducts'},\n]\n\ndef create_dag(config: dict) -> DAG:\n    \"\"\"Factory function to create DAGs from config\"\"\"\n\n    dag_id = f\"etl_{config['name']}\"\n\n    default_args = {\n        'owner': 'data-team',\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    }\n\n    dag = DAG(\n        dag_id=dag_id,\n        default_args=default_args,\n        schedule=config['schedule'],\n        start_date=datetime(2024, 1, 1),\n        catchup=False,\n        tags=['etl', 'dynamic', config['name']],\n    )\n\n    with dag:\n        def extract_fn(source, **context):\n            print(f\"Extracting from {source} for {context['ds']}\")\n\n        def transform_fn(**context):\n            print(f\"Transforming data for {context['ds']}\")\n\n        def load_fn(table_name, **context):\n            print(f\"Loading to {table_name} for {context['ds']}\")\n\n        extract = PythonOperator(\n            task_id='extract',\n            python_callable=extract_fn,\n            op_kwargs={'source': config['source']},\n        )\n\n        transform = PythonOperator(\n            task_id='transform',\n            python_callable=transform_fn,\n        )\n\n        load = PythonOperator(\n            task_id='load',\n            python_callable=load_fn,\n            op_kwargs={'table_name': config['name']},\n        )\n\n        extract >> transform >> load\n\n    return dag\n\n# Generate DAGs\nfor config in PIPELINE_CONFIGS:\n    globals()[f\"dag_{config['name']}\"] = create_dag(config)\n```\n\n### Pattern 3: Branching and Conditional Logic\n\n```python\n# dags\u002Fbranching_example.py\nfrom airflow.decorators import dag, task\nfrom airflow.operators.python import BranchPythonOperator\nfrom airflow.operators.empty import EmptyOperator\nfrom airflow.utils.trigger_rule import TriggerRule\n\n@dag(\n    dag_id='branching_pipeline',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n)\ndef branching_pipeline():\n\n    @task()\n    def check_data_quality() -> dict:\n        \"\"\"Check data quality and return metrics\"\"\"\n        quality_score = 0.95  # Simulated\n        return {'score': quality_score, 'rows': 10000}\n\n    def choose_branch(**context) -> str:\n        \"\"\"Determine which branch to execute\"\"\"\n        ti = context['ti']\n        metrics = ti.xcom_pull(task_ids='check_data_quality')\n\n        if metrics['score'] >= 0.9:\n            return 'high_quality_path'\n        elif metrics['score'] >= 0.7:\n            return 'medium_quality_path'\n        else:\n            return 'low_quality_path'\n\n    quality_check = check_data_quality()\n\n    branch = BranchPythonOperator(\n        task_id='branch',\n        python_callable=choose_branch,\n    )\n\n    high_quality = EmptyOperator(task_id='high_quality_path')\n    medium_quality = EmptyOperator(task_id='medium_quality_path')\n    low_quality = EmptyOperator(task_id='low_quality_path')\n\n    # Join point - runs after any branch completes\n    join = EmptyOperator(\n        task_id='join',\n        trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,\n    )\n\n    quality_check >> branch >> [high_quality, medium_quality, low_quality] >> join\n\nbranching_pipeline()\n```\n\n### Pattern 4: Sensors and External Dependencies\n\n```python\n# dags\u002Fsensor_patterns.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.sensors.filesystem import FileSensor\nfrom airflow.providers.amazon.aws.sensors.s3 import S3KeySensor\nfrom airflow.sensors.external_task import ExternalTaskSensor\nfrom airflow.operators.python import PythonOperator\n\nwith DAG(\n    dag_id='sensor_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n) as dag:\n\n    # Wait for file on S3\n    wait_for_file = S3KeySensor(\n        task_id='wait_for_s3_file',\n        bucket_name='data-lake',\n        bucket_key='raw\u002F{{ ds }}\u002Fdata.parquet',\n        aws_conn_id='aws_default',\n        timeout=60 * 60 * 2,  # 2 hours\n        poke_interval=60 * 5,  # Check every 5 minutes\n        mode='reschedule',  # Free up worker slot while waiting\n    )\n\n    # Wait for another DAG to complete\n    wait_for_upstream = ExternalTaskSensor(\n        task_id='wait_for_upstream_dag',\n        external_dag_id='upstream_etl',\n        external_task_id='final_task',\n        execution_date_fn=lambda dt: dt,  # Same execution date\n        timeout=60 * 60 * 3,\n        mode='reschedule',\n    )\n\n    # Custom sensor using @task.sensor decorator\n    @task.sensor(poke_interval=60, timeout=3600, mode='reschedule')\n    def wait_for_api() -> PokeReturnValue:\n        \"\"\"Custom sensor for API availability\"\"\"\n        import requests\n\n        response = requests.get('https:\u002F\u002Fapi.example.com\u002Fhealth')\n        is_done = response.status_code == 200\n\n        return PokeReturnValue(is_done=is_done, xcom_value=response.json())\n\n    api_ready = wait_for_api()\n\n    def process_data(**context):\n        api_result = context['ti'].xcom_pull(task_ids='wait_for_api')\n        print(f\"API returned: {api_result}\")\n\n    process = PythonOperator(\n        task_id='process',\n        python_callable=process_data,\n    )\n\n    [wait_for_file, wait_for_upstream, api_ready] >> process\n```\n\n### Pattern 5: Error Handling and Alerts\n\n```python\n# dags\u002Ferror_handling.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.utils.trigger_rule import TriggerRule\nfrom airflow.models import Variable\n\ndef task_failure_callback(context):\n    \"\"\"Callback on task failure\"\"\"\n    task_instance = context['task_instance']\n    exception = context.get('exception')\n\n    # Send to Slack\u002FPagerDuty\u002Fetc\n    message = f\"\"\"\n    Task Failed!\n    DAG: {task_instance.dag_id}\n    Task: {task_instance.task_id}\n    Execution Date: {context['ds']}\n    Error: {exception}\n    Log URL: {task_instance.log_url}\n    \"\"\"\n    # send_slack_alert(message)\n    print(message)\n\ndef dag_failure_callback(context):\n    \"\"\"Callback on DAG failure\"\"\"\n    # Aggregate failures, send summary\n    pass\n\nwith DAG(\n    dag_id='error_handling_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    on_failure_callback=dag_failure_callback,\n    default_args={\n        'on_failure_callback': task_failure_callback,\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    },\n) as dag:\n\n    def might_fail(**context):\n        import random\n        if random.random() \u003C 0.3:\n            raise ValueError(\"Random failure!\")\n        return \"Success\"\n\n    risky_task = PythonOperator(\n        task_id='risky_task',\n        python_callable=might_fail,\n    )\n\n    def cleanup(**context):\n        \"\"\"Cleanup runs regardless of upstream failures\"\"\"\n        print(\"Cleaning up...\")\n\n    cleanup_task = PythonOperator(\n        task_id='cleanup',\n        python_callable=cleanup,\n        trigger_rule=TriggerRule.ALL_DONE,  # Run even if upstream fails\n    )\n\n    def notify_success(**context):\n        \"\"\"Only runs if all upstream succeeded\"\"\"\n        print(\"All tasks succeeded!\")\n\n    success_notification = PythonOperator(\n        task_id='notify_success',\n        python_callable=notify_success,\n        trigger_rule=TriggerRule.ALL_SUCCESS,\n    )\n\n    risky_task >> [cleanup_task, success_notification]\n```\n\n### Pattern 6: Testing DAGs\n\n```python\n# tests\u002Ftest_dags.py\nimport pytest\nfrom datetime import datetime\nfrom airflow.models import DagBag\n\n@pytest.fixture\ndef dagbag():\n    return DagBag(dag_folder='dags\u002F', include_examples=False)\n\ndef test_dag_loaded(dagbag):\n    \"\"\"Test that all DAGs load without errors\"\"\"\n    assert len(dagbag.import_errors) == 0, f\"DAG import errors: {dagbag.import_errors}\"\n\ndef test_dag_structure(dagbag):\n    \"\"\"Test specific DAG structure\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    assert dag is not None\n    assert len(dag.tasks) == 3\n    assert dag.schedule_interval == '0 6 * * *'\n\ndef test_task_dependencies(dagbag):\n    \"\"\"Test task dependencies are correct\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    extract_task = dag.get_task('extract')\n    assert 'start' in [t.task_id for t in extract_task.upstream_list]\n    assert 'end' in [t.task_id for t in extract_task.downstream_list]\n\ndef test_dag_integrity(dagbag):\n    \"\"\"Test DAG has no cycles and is valid\"\"\"\n    for dag_id, dag in dagbag.dags.items():\n        assert dag.test_cycle() is None, f\"Cycle detected in {dag_id}\"\n\n# Test individual task logic\ndef test_extract_function():\n    \"\"\"Unit test for extract function\"\"\"\n    from dags.example_dag import extract_data\n\n    result = extract_data(ds='2024-01-01')\n    assert 'records' in result\n    assert isinstance(result['records'], int)\n```\n\n## Project Structure\n\n```\nairflow\u002F\n├── dags\u002F\n│   ├── __init__.py\n│   ├── common\u002F\n│   │   ├── __init__.py\n│   │   ├── operators.py    # Custom operators\n│   │   ├── sensors.py      # Custom sensors\n│   │   └── callbacks.py    # Alert callbacks\n│   ├── etl\u002F\n│   │   ├── customers.py\n│   │   └── orders.py\n│   └── ml\u002F\n│       └── training.py\n├── plugins\u002F\n│   └── custom_plugin.py\n├── tests\u002F\n│   ├── __init__.py\n│   ├── test_dags.py\n│   └── test_operators.py\n├── docker-compose.yml\n└── requirements.txt\n```\n\n## Best Practices\n\n### Do's\n\n- **Use TaskFlow API** - Cleaner code, automatic XCom\n- **Set timeouts** - Prevent zombie tasks\n- **Use `mode='reschedule'`** - For sensors, free up workers\n- **Test DAGs** - Unit tests and integration tests\n- **Idempotent tasks** - Safe to retry\n\n### Don'ts\n\n- **Don't use `depends_on_past=True`** - Creates bottlenecks\n- **Don't hardcode dates** - Use `{{ ds }}` macros\n- **Don't use global state** - Tasks should be stateless\n- **Don't skip catchup blindly** - Understand implications\n- **Don't put heavy logic in DAG file** - Import from modules\n","# Apache Airflow DAG Patterns\n\nProduction-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.\n\n## When to Use This Skill\n\n- Creating data pipeline orchestration with Airflow\n- Designing DAG structures and dependencies\n- Implementing custom operators and sensors\n- Testing Airflow DAGs locally\n- Setting up Airflow in production\n- Debugging failed DAG runs\n\n## Core Concepts\n\n### 1. DAG Design Principles\n\n| Principle       | Description                         |\n| --------------- | ----------------------------------- |\n| **Idempotent**  | Running twice produces same result  |\n| **Atomic**      | Tasks succeed or fail completely    |\n| **Incremental** | Process only new\u002Fchanged data       |\n| **Observable**  | Logs, metrics, alerts at every step |\n\n### 2. Task Dependencies\n\n```python\n# Linear\ntask1 >> task2 >> task3\n\n# Fan-out\ntask1 >> [task2, task3, task4]\n\n# Fan-in\n[task1, task2, task3] >> task4\n\n# Complex\ntask1 >> task2 >> task4\ntask1 >> task3 >> task4\n```\n\n## Quick Start\n\n```python\n# dags\u002Fexample_dag.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.operators.empty import EmptyOperator\n\ndefault_args = {\n    'owner': 'data-team',\n    'depends_on_past': False,\n    'email_on_failure': True,\n    'email_on_retry': False,\n    'retries': 3,\n    'retry_delay': timedelta(minutes=5),\n    'retry_exponential_backoff': True,\n    'max_retry_delay': timedelta(hours=1),\n}\n\nwith DAG(\n    dag_id='example_etl',\n    default_args=default_args,\n    description='Example ETL pipeline',\n    schedule='0 6 * * *',  # Daily at 6 AM\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'example'],\n    max_active_runs=1,\n) as dag:\n\n    start = EmptyOperator(task_id='start')\n\n    def extract_data(**context):\n        execution_date = context['ds']\n        # Extract logic here\n        return {'records': 1000}\n\n    extract = PythonOperator(\n        task_id='extract',\n        python_callable=extract_data,\n    )\n\n    end = EmptyOperator(task_id='end')\n\n    start >> extract >> end\n```\n\n## Patterns\n\n### Pattern 1: TaskFlow API (Airflow 2.0+)\n\n```python\n# dags\u002Ftaskflow_example.py\nfrom datetime import datetime\nfrom airflow.decorators import dag, task\nfrom airflow.models import Variable\n\n@dag(\n    dag_id='taskflow_etl',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'taskflow'],\n)\ndef taskflow_etl():\n    \"\"\"ETL pipeline using TaskFlow API\"\"\"\n\n    @task()\n    def extract(source: str) -> dict:\n        \"\"\"Extract data from source\"\"\"\n        import pandas as pd\n\n        df = pd.read_csv(f's3:\u002F\u002Fbucket\u002F{source}\u002F{{ ds }}.csv')\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def transform(extracted: dict) -> dict:\n        \"\"\"Transform extracted data\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(extracted['data'])\n        df['processed_at'] = datetime.now()\n        df = df.dropna()\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def load(transformed: dict, target: str):\n        \"\"\"Load data to target\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(transformed['data'])\n        df.to_parquet(f's3:\u002F\u002Fbucket\u002F{target}\u002F{{ ds }}.parquet')\n        return transformed['rows']\n\n    @task()\n    def notify(rows_loaded: int):\n        \"\"\"Send notification\"\"\"\n        print(f'Loaded {rows_loaded} rows')\n\n    # Define dependencies with XCom passing\n    extracted = extract(source='raw_data')\n    transformed = transform(extracted)\n    loaded = load(transformed, target='processed_data')\n    notify(loaded)\n\n# Instantiate the DAG\ntaskflow_etl()\n```\n\n### Pattern 2: Dynamic DAG Generation\n\n```python\n# dags\u002Fdynamic_dag_factory.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.models import Variable\nimport json\n\n# Configuration for multiple similar pipelines\nPIPELINE_CONFIGS = [\n    {'name': 'customers', 'schedule': '@daily', 'source': 's3:\u002F\u002Fraw\u002Fcustomers'},\n    {'name': 'orders', 'schedule': '@hourly', 'source': 's3:\u002F\u002Fraw\u002Forders'},\n    {'name': 'products', 'schedule': '@weekly', 'source': 's3:\u002F\u002Fraw\u002Fproducts'},\n]\n\ndef create_dag(config: dict) -> DAG:\n    \"\"\"Factory function to create DAGs from config\"\"\"\n\n    dag_id = f\"etl_{config['name']}\"\n\n    default_args = {\n        'owner': 'data-team',\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    }\n\n    dag = DAG(\n        dag_id=dag_id,\n        default_args=default_args,\n        schedule=config['schedule'],\n        start_date=datetime(2024, 1, 1),\n        catchup=False,\n        tags=['etl', 'dynamic', config['name']],\n    )\n\n    with dag:\n        def extract_fn(source, **context):\n            print(f\"Extracting from {source} for {context['ds']}\")\n\n        def transform_fn(**context):\n            print(f\"Transforming data for {context['ds']}\")\n\n        def load_fn(table_name, **context):\n            print(f\"Loading to {table_name} for {context['ds']}\")\n\n        extract = PythonOperator(\n            task_id='extract',\n            python_callable=extract_fn,\n            op_kwargs={'source': config['source']},\n        )\n\n        transform = PythonOperator(\n            task_id='transform',\n            python_callable=transform_fn,\n        )\n\n        load = PythonOperator(\n            task_id='load',\n            python_callable=load_fn,\n            op_kwargs={'table_name': config['name']},\n        )\n\n        extract >> transform >> load\n\n    return dag\n\n# Generate DAGs\nfor config in PIPELINE_CONFIGS:\n    globals()[f\"dag_{config['name']}\"] = create_dag(config)\n```\n\n### Pattern 3: Branching and Conditional Logic\n\n```python\n# dags\u002Fbranching_example.py\nfrom airflow.decorators import dag, task\nfrom airflow.operators.python import BranchPythonOperator\nfrom airflow.operators.empty import EmptyOperator\nfrom airflow.utils.trigger_rule import TriggerRule\n\n@dag(\n    dag_id='branching_pipeline',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n)\ndef branching_pipeline():\n\n    @task()\n    def check_data_quality() -> dict:\n        \"\"\"Check data quality and return metrics\"\"\"\n        quality_score = 0.95  # Simulated\n        return {'score': quality_score, 'rows': 10000}\n\n    def choose_branch(**context) -> str:\n        \"\"\"Determine which branch to execute\"\"\"\n        ti = context['ti']\n        metrics = ti.xcom_pull(task_ids='check_data_quality')\n\n        if metrics['score'] >= 0.9:\n            return 'high_quality_path'\n        elif metrics['score'] >= 0.7:\n            return 'medium_quality_path'\n        else:\n            return 'low_quality_path'\n\n    quality_check = check_data_quality()\n\n    branch = BranchPythonOperator(\n        task_id='branch',\n        python_callable=choose_branch,\n    )\n\n    high_quality = EmptyOperator(task_id='high_quality_path')\n    medium_quality = EmptyOperator(task_id='medium_quality_path')\n    low_quality = EmptyOperator(task_id='low_quality_path')\n\n    # Join point - runs after any branch completes\n    join = EmptyOperator(\n        task_id='join',\n        trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,\n    )\n\n    quality_check >> branch >> [high_quality, medium_quality, low_quality] >> join\n\nbranching_pipeline()\n```\n\n### Pattern 4: Sensors and External Dependencies\n\n```python\n# dags\u002Fsensor_patterns.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.sensors.filesystem import FileSensor\nfrom airflow.providers.amazon.aws.sensors.s3 import S3KeySensor\nfrom airflow.sensors.external_task import ExternalTaskSensor\nfrom airflow.operators.python import PythonOperator\n\nwith DAG(\n    dag_id='sensor_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n) as dag:\n\n    # Wait for file on S3\n    wait_for_file = S3KeySensor(\n        task_id='wait_for_s3_file',\n        bucket_name='data-lake',\n        bucket_key='raw\u002F{{ ds }}\u002Fdata.parquet',\n        aws_conn_id='aws_default',\n        timeout=60 * 60 * 2,  # 2 hours\n        poke_interval=60 * 5,  # Check every 5 minutes\n        mode='reschedule',  # Free up worker slot while waiting\n    )\n\n    # Wait for another DAG to complete\n    wait_for_upstream = ExternalTaskSensor(\n        task_id='wait_for_upstream_dag',\n        external_dag_id='upstream_etl',\n        external_task_id='final_task',\n        execution_date_fn=lambda dt: dt,  # Same execution date\n        timeout=60 * 60 * 3,\n        mode='reschedule',\n    )\n\n    # Custom sensor using @task.sensor decorator\n    @task.sensor(poke_interval=60, timeout=3600, mode='reschedule')\n    def wait_for_api() -> PokeReturnValue:\n        \"\"\"Custom sensor for API availability\"\"\"\n        import requests\n\n        response = requests.get('https:\u002F\u002Fapi.example.com\u002Fhealth')\n        is_done = response.status_code == 200\n\n        return PokeReturnValue(is_done=is_done, xcom_value=response.json())\n\n    api_ready = wait_for_api()\n\n    def process_data(**context):\n        api_result = context['ti'].xcom_pull(task_ids='wait_for_api')\n        print(f\"API returned: {api_result}\")\n\n    process = PythonOperator(\n        task_id='process',\n        python_callable=process_data,\n    )\n\n    [wait_for_file, wait_for_upstream, api_ready] >> process\n```\n\n### Pattern 5: Error Handling and Alerts\n\n```python\n# dags\u002Ferror_handling.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.utils.trigger_rule import TriggerRule\nfrom airflow.models import Variable\n\ndef task_failure_callback(context):\n    \"\"\"Callback on task failure\"\"\"\n    task_instance = context['task_instance']\n    exception = context.get('exception')\n\n    # Send to Slack\u002FPagerDuty\u002Fetc\n    message = f\"\"\"\n    Task Failed!\n    DAG: {task_instance.dag_id}\n    Task: {task_instance.task_id}\n    Execution Date: {context['ds']}\n    Error: {exception}\n    Log URL: {task_instance.log_url}\n    \"\"\"\n    # send_slack_alert(message)\n    print(message)\n\ndef dag_failure_callback(context):\n    \"\"\"Callback on DAG failure\"\"\"\n    # Aggregate failures, send summary\n    pass\n\nwith DAG(\n    dag_id='error_handling_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    on_failure_callback=dag_failure_callback,\n    default_args={\n        'on_failure_callback': task_failure_callback,\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    },\n) as dag:\n\n    def might_fail(**context):\n        import random\n        if random.random() \u003C 0.3:\n            raise ValueError(\"Random failure!\")\n        return \"Success\"\n\n    risky_task = PythonOperator(\n        task_id='risky_task',\n        python_callable=might_fail,\n    )\n\n    def cleanup(**context):\n        \"\"\"Cleanup runs regardless of upstream failures\"\"\"\n        print(\"Cleaning up...\")\n\n    cleanup_task = PythonOperator(\n        task_id='cleanup',\n        python_callable=cleanup,\n        trigger_rule=TriggerRule.ALL_DONE,  # Run even if upstream fails\n    )\n\n    def notify_success(**context):\n        \"\"\"Only runs if all upstream succeeded\"\"\"\n        print(\"All tasks succeeded!\")\n\n    success_notification = PythonOperator(\n        task_id='notify_success',\n        python_callable=notify_success,\n        trigger_rule=TriggerRule.ALL_SUCCESS,\n    )\n\n    risky_task >> [cleanup_task, success_notification]\n```\n\n### Pattern 6: Testing DAGs\n\n```python\n# tests\u002Ftest_dags.py\nimport pytest\nfrom datetime import datetime\nfrom airflow.models import DagBag\n\n@pytest.fixture\ndef dagbag():\n    return DagBag(dag_folder='dags\u002F', include_examples=False)\n\ndef test_dag_loaded(dagbag):\n    \"\"\"Test that all DAGs load without errors\"\"\"\n    assert len(dagbag.import_errors) == 0, f\"DAG import errors: {dagbag.import_errors}\"\n\ndef test_dag_structure(dagbag):\n    \"\"\"Test specific DAG structure\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    assert dag is not None\n    assert len(dag.tasks) == 3\n    assert dag.schedule_interval == '0 6 * * *'\n\ndef test_task_dependencies(dagbag):\n    \"\"\"Test task dependencies are correct\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    extract_task = dag.get_task('extract')\n    assert 'start' in [t.task_id for t in extract_task.upstream_list]\n    assert 'end' in [t.task_id for t in extract_task.downstream_list]\n\ndef test_dag_integrity(dagbag):\n    \"\"\"Test DAG has no cycles and is valid\"\"\"\n    for dag_id, dag in dagbag.dags.items():\n        assert dag.test_cycle() is None, f\"Cycle detected in {dag_id}\"\n\n# Test individual task logic\ndef test_extract_function():\n    \"\"\"Unit test for extract function\"\"\"\n    from dags.example_dag import extract_data\n\n    result = extract_data(ds='2024-01-01')\n    assert 'records' in result\n    assert isinstance(result['records'], int)\n```\n\n## Project Structure\n\n```\nairflow\u002F\n├── dags\u002F\n│   ├── __init__.py\n│   ├── common\u002F\n│   │   ├── __init__.py\n│   │   ├── operators.py    # Custom operators\n│   │   ├── sensors.py      # Custom sensors\n│   │   └── callbacks.py    # Alert callbacks\n│   ├── etl\u002F\n│   │   ├── customers.py\n│   │   └── orders.py\n│   └── ml\u002F\n│       └── training.py\n├── plugins\u002F\n│   └── custom_plugin.py\n├── tests\u002F\n│   ├── __init__.py\n│   ├── test_dags.py\n│   └── test_operators.py\n├── docker-compose.yml\n└── requirements.txt\n```\n\n## Best Practices\n\n### Do's\n\n- **Use TaskFlow API** - Cleaner code, automatic XCom\n- **Set timeouts** - Prevent zombie tasks\n- **Use `mode='reschedule'`** - For sensors, free up workers\n- **Test DAGs** - Unit tests and integration tests\n- **Idempotent tasks** - Safe to retry\n\n### Don'ts\n\n- **Don't use `depends_on_past=True`** - Creates bottlenecks\n- **Don't hardcode dates** - Use `{{ ds }}` macros\n- **Don't use global state** - Tasks should be stateless\n- **Don't skip catchup blindly** - Understand implications\n- **Don't put heavy logic in DAG file** - Import from modules\n",{"name":5,"description":7},"main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-Plugin-插件协议\u002FYYC3-Plugin-插件协议-plugins-claude\u002Fdata-engineering\u002Fskills\u002Fairflow-dag-patterns",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill airflow-dag-patterns",{"tool":39,"command":40},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill airflow-dag-patterns",{"tool":42,"command":43},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill airflow-dag-patterns","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Plugin-插件协议\u002FYYC3-Plugin-插件协议-plugins-claude\u002Fdata-engineering\u002Fskills\u002Fairflow-dag-patterns\u002FSKILL.md",true,[],[48,55,62,69,76,83],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]