[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7DM779PYQSHQKJ6RTKMN54V":3,"skill-same:01M7DM779PYQSHQKJ6RTKMN54V":46},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M7DM779PYQSHQKJ6RTKMN54V","dag-orchestration-patterns",null,"Guide to data pipeline orchestration: Airflow DAG design patterns, dependency management, retry strategies, idempotency guarantees and backfill strategies. Use this skill for pipeline scheduling: 'Airflow DAG', 'DAG design', 'dependencies', 'retry strategy', 'idempotency', 'backfill', 'pipeline orchestration', 'Dagster', 'Prefect'. It strengthens the DAG design capability of scheduler-engineer. Defining data quality rules and monitoring dashboards are outside the scope of this skill.","danielnguyenfinhub\u002Ffinhub-harness","danielnguyenfinhub","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F264356854?v=4","01M7DM55XETDR9HF92FWC28SBN","Daniel Nguyen","01M7DM55YGJYBWW488JZSQYEJ4",1,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Fpacks\u002Ffinhub-ops-pack\u002Fskills\u002Fdag-orchestration-patterns\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: dag-orchestration-patterns\ndescription: \"Guide to data pipeline orchestration: Airflow DAG design patterns, dependency management, retry strategies, idempotency guarantees and backfill strategies. Use this skill for pipeline scheduling: 'Airflow DAG', 'DAG design', 'dependencies', 'retry strategy', 'idempotency', 'backfill', 'pipeline orchestration', 'Dagster', 'Prefect'. It strengthens the DAG design capability of scheduler-engineer. Defining data quality rules and monitoring dashboards are outside the scope of this skill.\"\n---\n\n# DAG Orchestration Patterns — Pipeline Orchestration Pattern Guide\n\nAirflow-centred DAG design patterns and operating strategies.\n\n## DAG Design Patterns\n\n### 1. Extract-Load-Transform (ELT) Pattern\n\n```python\nwith DAG(\"elt_orders\", schedule=\"0 2 * * *\", catchup=False) as dag:\n    extract = PythonOperator(task_id=\"extract\", python_callable=extract_orders)\n    load_raw = PythonOperator(task_id=\"load_raw\", python_callable=load_to_raw)\n    transform = DbtOperator(task_id=\"transform\", select=\"orders\")\n    quality = PythonOperator(task_id=\"quality_check\", python_callable=run_checks)\n    notify = SlackOperator(task_id=\"notify\", trigger_rule=\"all_done\")\n\n    extract >> load_raw >> transform >> quality >> notify\n```\n\n### 2. Fan-out\u002FFan-in Pattern\n\n```python\n# Parallel extraction from multiple sources → combined transformation\nsources = [\"mysql\", \"postgres\", \"api\"]\nextract_tasks = [\n    PythonOperator(task_id=f\"extract_{src}\", python_callable=extract, op_args=[src])\n    for src in sources\n]\nmerge = PythonOperator(task_id=\"merge_all\", python_callable=merge_sources)\ntransform = PythonOperator(task_id=\"transform\", python_callable=transform_data)\n\nextract_tasks >> merge >> transform\n```\n\n### 3. Sensor-Based Event Waiting\n\n```python\nwait_for_data = S3KeySensor(\n    task_id=\"wait_for_file\",\n    bucket_name=\"data-lake\",\n    bucket_key=\"raw\u002Forders\u002F{{ ds }}\u002Fdata.parquet\",\n    timeout=3600,  # wait up to 1 hour\n    poke_interval=60,\n    mode=\"reschedule\"  # release the worker while waiting\n)\n```\n\n## Idempotency Patterns\n\n### Partition Replacement (Most Recommended)\n\n```sql\n-- Replace the entire date partition (idempotent)\nDELETE FROM analytics.orders WHERE date_partition = '{{ ds }}';\nINSERT INTO analytics.orders\nSELECT * FROM staging.orders WHERE date_partition = '{{ ds }}';\n```\n\n### MERGE\u002FUPSERT\n\n```sql\nMERGE INTO target AS t\nUSING source AS s ON t.id = s.id\nWHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.updated_at = CURRENT_TIMESTAMP\nWHEN NOT MATCHED THEN INSERT (id, amount, created_at) VALUES (s.id, s.amount, CURRENT_TIMESTAMP);\n```\n\n### Idempotency Checklist\n\n- [ ] Does running the same DAG twice produce the same result?\n- [ ] Does it use the date parameter (`{{ ds }}`) to limit the range?\n- [ ] Does it clear existing data before the INSERT?\n- [ ] Does it use a unique request ID for external API calls?\n\n## Retry Strategy\n\n```python\ndefault_args = {\n    \"retries\": 3,\n    \"retry_delay\": timedelta(minutes=5),\n    \"retry_exponential_backoff\": True,\n    \"max_retry_delay\": timedelta(minutes=60),\n    \"execution_timeout\": timedelta(hours=2),\n}\n```\n\n### Retry Settings by Task\n\n| Task type | Retries | Wait time | Reason |\n|----------|-----------|----------|------|\n| DB extraction | 3 | 5 minutes, exponential backoff | Transient connection problems |\n| API call | 5 | 30 seconds, exponential backoff | Rate limits, network |\n| Transformation (SQL) | 1 | Immediate | Retrying a logic error is pointless |\n| File upload | 3 | 1 minute | Unstable network |\n\n## Backfill Strategy\n\n```python\n# Safe backfill settings\ndag = DAG(\n    \"daily_orders\",\n    schedule=\"0 2 * * *\",\n    start_date=datetime(2024, 1, 1),\n    catchup=False,  # disable automatic backfill\n    max_active_runs=1,  # prevent concurrent runs\n)\n\n# Manual backfill from the CLI\n# airflow dags backfill daily_orders -s 2024-01-01 -e 2024-01-31\n```\n\n### Backfill Cautions\n\n| Caution | Description | Response |\n|------|------|------|\n| Concurrent runs | The same data range is processed twice | `max_active_runs=1` |\n| API rate limits | Large requests for historical data | Limit batch size, insert sleeps |\n| Resource contention | DB\u002Fcluster overload | Limit parallelism, spread runs over time |\n| Data consistency | Historical data had a different structure | Schema evolution handling logic |\n\n## Dependency Patterns\n\n### Cross-DAG Dependency\n\n```python\n# Wait for DAG A to complete\nwait_for_upstream = ExternalTaskSensor(\n    task_id=\"wait_for_dag_a\",\n    external_dag_id=\"dag_a\",\n    external_task_id=\"final_task\",\n    execution_delta=timedelta(hours=0),\n    timeout=3600,\n    mode=\"reschedule\"\n)\n```\n\n### Dataset-Based Dependency (Airflow 2.4+)\n\n```python\n# Producer DAG\norders_dataset = Dataset(\"s3:\u002F\u002Fdatalake\u002Forders\u002F\")\n\nwith DAG(\"produce_orders\", schedule=\"0 2 * * *\") as dag:\n    produce = PythonOperator(\n        task_id=\"produce\", outlets=[orders_dataset]\n    )\n\n# Consumer DAG — triggered automatically\nwith DAG(\"consume_orders\", schedule=[orders_dataset]) as dag:\n    consume = PythonOperator(task_id=\"consume\", ...)\n```\n\n## Alerting Strategy\n\n```python\ndef failure_callback(context):\n    task = context[\"task_instance\"]\n    dag_id = context[\"dag\"].dag_id\n    execution_date = context[\"execution_date\"]\n    message = f\"FAILED: {dag_id}\u002F{task.task_id} at {execution_date}\"\n    send_slack(message)\n\ndefault_args = {\n    \"on_failure_callback\": failure_callback,\n    \"on_retry_callback\": retry_callback,\n}\n```\n\n| Event | Channel | Audience |\n|--------|------|------|\n| P0 task failure | Slack + PagerDuty | On-call engineer |\n| SLA missed | Slack | Data team |\n| Retry occurred | Slack (informational) | Monitoring channel |\n| Backfill complete | Slack | Requester |\n","# DAG Orchestration Patterns — Pipeline Orchestration Pattern Guide\n\nAirflow-centred DAG design patterns and operating strategies.\n\n## DAG Design Patterns\n\n### 1. Extract-Load-Transform (ELT) Pattern\n\n```python\nwith DAG(\"elt_orders\", schedule=\"0 2 * * *\", catchup=False) as dag:\n    extract = PythonOperator(task_id=\"extract\", python_callable=extract_orders)\n    load_raw = PythonOperator(task_id=\"load_raw\", python_callable=load_to_raw)\n    transform = DbtOperator(task_id=\"transform\", select=\"orders\")\n    quality = PythonOperator(task_id=\"quality_check\", python_callable=run_checks)\n    notify = SlackOperator(task_id=\"notify\", trigger_rule=\"all_done\")\n\n    extract >> load_raw >> transform >> quality >> notify\n```\n\n### 2. Fan-out\u002FFan-in Pattern\n\n```python\n# Parallel extraction from multiple sources → combined transformation\nsources = [\"mysql\", \"postgres\", \"api\"]\nextract_tasks = [\n    PythonOperator(task_id=f\"extract_{src}\", python_callable=extract, op_args=[src])\n    for src in sources\n]\nmerge = PythonOperator(task_id=\"merge_all\", python_callable=merge_sources)\ntransform = PythonOperator(task_id=\"transform\", python_callable=transform_data)\n\nextract_tasks >> merge >> transform\n```\n\n### 3. Sensor-Based Event Waiting\n\n```python\nwait_for_data = S3KeySensor(\n    task_id=\"wait_for_file\",\n    bucket_name=\"data-lake\",\n    bucket_key=\"raw\u002Forders\u002F{{ ds }}\u002Fdata.parquet\",\n    timeout=3600,  # wait up to 1 hour\n    poke_interval=60,\n    mode=\"reschedule\"  # release the worker while waiting\n)\n```\n\n## Idempotency Patterns\n\n### Partition Replacement (Most Recommended)\n\n```sql\n-- Replace the entire date partition (idempotent)\nDELETE FROM analytics.orders WHERE date_partition = '{{ ds }}';\nINSERT INTO analytics.orders\nSELECT * FROM staging.orders WHERE date_partition = '{{ ds }}';\n```\n\n### MERGE\u002FUPSERT\n\n```sql\nMERGE INTO target AS t\nUSING source AS s ON t.id = s.id\nWHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.updated_at = CURRENT_TIMESTAMP\nWHEN NOT MATCHED THEN INSERT (id, amount, created_at) VALUES (s.id, s.amount, CURRENT_TIMESTAMP);\n```\n\n### Idempotency Checklist\n\n- [ ] Does running the same DAG twice produce the same result?\n- [ ] Does it use the date parameter (`{{ ds }}`) to limit the range?\n- [ ] Does it clear existing data before the INSERT?\n- [ ] Does it use a unique request ID for external API calls?\n\n## Retry Strategy\n\n```python\ndefault_args = {\n    \"retries\": 3,\n    \"retry_delay\": timedelta(minutes=5),\n    \"retry_exponential_backoff\": True,\n    \"max_retry_delay\": timedelta(minutes=60),\n    \"execution_timeout\": timedelta(hours=2),\n}\n```\n\n### Retry Settings by Task\n\n| Task type | Retries | Wait time | Reason |\n|----------|-----------|----------|------|\n| DB extraction | 3 | 5 minutes, exponential backoff | Transient connection problems |\n| API call | 5 | 30 seconds, exponential backoff | Rate limits, network |\n| Transformation (SQL) | 1 | Immediate | Retrying a logic error is pointless |\n| File upload | 3 | 1 minute | Unstable network |\n\n## Backfill Strategy\n\n```python\n# Safe backfill settings\ndag = DAG(\n    \"daily_orders\",\n    schedule=\"0 2 * * *\",\n    start_date=datetime(2024, 1, 1),\n    catchup=False,  # disable automatic backfill\n    max_active_runs=1,  # prevent concurrent runs\n)\n\n# Manual backfill from the CLI\n# airflow dags backfill daily_orders -s 2024-01-01 -e 2024-01-31\n```\n\n### Backfill Cautions\n\n| Caution | Description | Response |\n|------|------|------|\n| Concurrent runs | The same data range is processed twice | `max_active_runs=1` |\n| API rate limits | Large requests for historical data | Limit batch size, insert sleeps |\n| Resource contention | DB\u002Fcluster overload | Limit parallelism, spread runs over time |\n| Data consistency | Historical data had a different structure | Schema evolution handling logic |\n\n## Dependency Patterns\n\n### Cross-DAG Dependency\n\n```python\n# Wait for DAG A to complete\nwait_for_upstream = ExternalTaskSensor(\n    task_id=\"wait_for_dag_a\",\n    external_dag_id=\"dag_a\",\n    external_task_id=\"final_task\",\n    execution_delta=timedelta(hours=0),\n    timeout=3600,\n    mode=\"reschedule\"\n)\n```\n\n### Dataset-Based Dependency (Airflow 2.4+)\n\n```python\n# Producer DAG\norders_dataset = Dataset(\"s3:\u002F\u002Fdatalake\u002Forders\u002F\")\n\nwith DAG(\"produce_orders\", schedule=\"0 2 * * *\") as dag:\n    produce = PythonOperator(\n        task_id=\"produce\", outlets=[orders_dataset]\n    )\n\n# Consumer DAG — triggered automatically\nwith DAG(\"consume_orders\", schedule=[orders_dataset]) as dag:\n    consume = PythonOperator(task_id=\"consume\", ...)\n```\n\n## Alerting Strategy\n\n```python\ndef failure_callback(context):\n    task = context[\"task_instance\"]\n    dag_id = context[\"dag\"].dag_id\n    execution_date = context[\"execution_date\"]\n    message = f\"FAILED: {dag_id}\u002F{task.task_id} at {execution_date}\"\n    send_slack(message)\n\ndefault_args = {\n    \"on_failure_callback\": failure_callback,\n    \"on_retry_callback\": retry_callback,\n}\n```\n\n| Event | Channel | Audience |\n|--------|------|------|\n| P0 task failure | Slack + PagerDuty | On-call engineer |\n| SLA missed | Slack | Data team |\n| Retry occurred | Slack (informational) | Monitoring channel |\n| Backfill complete | Slack | Requester |\n",{"name":5,"description":7},"main","dda161ff9daf0d9bccbdba26e34bbee0438a18b0","packs\u002Ffinhub-ops-pack\u002Fskills\u002Fdag-orchestration-patterns",0,"https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub","www.finhub.net.au","2026-11-12 17:07:32",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add danielnguyenfinhub\u002Ffinhub-harness --skill dag-orchestration-patterns",{"tool":38,"command":39},"bunx","bunx skills add danielnguyenfinhub\u002Ffinhub-harness --skill dag-orchestration-patterns",{"tool":41,"command":42},"pnpm","pnpm dlx skills add danielnguyenfinhub\u002Ffinhub-harness --skill dag-orchestration-patterns","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Fpacks\u002Ffinhub-ops-pack\u002Fskills\u002Fdag-orchestration-patterns\u002FSKILL.md",true,[],[47,54,61,67,73,80],{"skillNo":48,"name":49,"nameZh":6,"description":50,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":51,"categories":52,"keywords":53,"publishedAt":19},"01M7DMBP1MQ9X39BGJ9VF62M9P","linky","Fixture whose links and bundled-file mentions all resolve. Use to prove no false errors.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_ok\u002Fskills\u002Flinky\u002FSKILL.md",[],[],{"skillNo":55,"name":56,"nameZh":6,"description":57,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":19},"01M7DMBK70KXF6A3V1VDRHYGVT","dup-one","Second owner of the name. Use for the duplicate fixture.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fdup-two\u002FSKILL.md",[],[],{"skillNo":62,"name":56,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M7DMBFWK8HD4AWNK89KGE89P","First owner of the name. Use for the duplicate fixture.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fdup-one\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":19},"01M7DMBETX7172TFNBAA3BVREQ","nested","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fbroken\u002Fnested\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":19},"01M7DMBDCH5G61QVAB1QFMQA55","broken","Fixture with one defect per rule. Use to prove each rule fires.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fbroken\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7DMBAQHY0Z2PTYGCPM8H7MT","notes-style","House style for client notes. Use when writing or editing a client note.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_good\u002Fskills\u002Fnotes-style\u002FSKILL.md",[],[]]