etl
Extract, Transform, Load architecture, batch extraction, schema validation, idempotent upserts, and dead-letter queues
- License 许可
- MIT
- 最近更新
- 2026-11-09
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- etl
- risk
- low
- source
- https://github.com/Mr-Nobody-Anonymous/All-skills
- license
- MIT
- version
- 1.0.0
- category
- data-engineering
- description
- Extract, Transform, Load architecture, batch extraction, schema validation, idempotent upserts, and dead-letter queues
- imported_at
- 2026-09-20
- source_path
- awesome_skills/data-engineering/etl/SKILL.md
- source_repository
- Mr-Nobody-Anonymous/All-skills
- disable-model-invocation
- false
03 SKILL.md 正文
ETL Pipeline Architecture
Scope
ETL (Extract, Transform, Load) pipelines extract raw operational data from disparate sources, transform it according to business and analytical logic, and load clean tabular structures into target warehouses.
Architectural Principles & Invariants
- Idempotency & Replayability: Every pipeline run for a given time partition $[t_1, t_2]$ must yield identical state, regardless of rerun count. Implement via staging tables and atomic swaps:
MERGE INTO analytics.fact_orders t USING staging.orders_stg s ON t.order_id = s.order_id WHEN MATCHED THEN UPDATE SET ... WHEN NOT MATCHED THEN INSERT ...; - Schema Validation & Dead-Letter Queues (DLQ): Malformed rows failing type casting or schema contracts are directed to quarantine dead-letter sinks without halting pipeline execution.
- Incremental Extraction (CDC / Watermarking): Querying source tables using high-watermark timestamps:
WHERE updated_at > :last_watermark.
Tools & Standards
- Tools: Apache Airflow, dbt, Apache Spark, Singer, Airbyte.
04 同仓库更多 Skills
dei-assessment
Assess diversity, equity, and inclusion with metrics. TRIGGERS - Use when user needs help with dei-assessment related tasks.
dei-action-plan
Create DEI action plans with goals and initiatives. TRIGGERS - Use when user needs help with dei-action-plan related tasks.
culture-fit-assessment
Design culture add assessment processes. TRIGGERS - Use when user needs help with culture-fit-assessment related tasks.
cowork-to-code-bridge
Use an already-installed, independently verified cowork-to-code bridge to run narrowly approved actions on the user's own macOS, Linux, or WSL2 machine through a local file queue.
counter-offer-response
Draft counter-offer responses for retention. TRIGGERS - Use when user needs help with counter-offer-response related tasks.
continuous-feedback-system
Design continuous feedback systems. TRIGGERS - Use when user needs help with continuous-feedback-system related tasks.