data-pipelines
Build ETL/ELT and streaming pipelines with quality validation, schema evolution and anomaly correction. Use when building ETL/ELT pipelines, streaming data, or schema validation.
- License 许可
- MIT
- 最近更新
- 2026-10-02
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- data-pipelines
- group
- Data and AI
- description
- Build ETL/ELT and streaming pipelines with quality validation, schema evolution and anomaly correction. Use when building ETL/ELT pipelines, streaming data, or schema validation.
- last_reviewed
- 1788652800000
02 资源文件 2
- 其他data-pipelines.skill1.8 KB
- 参考文档evals/routing.yaml864 B
03 SKILL.md 正文
Data Pipelines
Architects reliable, scalable data pipelines from ingestion through transformation to serving, with built-in quality validation and schema governance.
Phased Workflow
Phase 1: Source Analysis & Architecture Selection
- Inventory data sources: APIs, databases, event streams, file drops, SaaS webhooks.
- Classify each pipeline: Batch (hourly/daily, high volume) vs Streaming (real-time, event-driven).
- Select architecture pattern: ELT (extract-load-transform in warehouse) vs ETL (transform before load).
Phase 2: Pipeline Implementation
- Design idempotent, retry-safe extraction with checkpoint/offset tracking.
- Implement transformations using dbt models (staging → intermediate → marts) or Spark jobs.
- Define schema evolution strategy: additive-only columns, versioned schemas, backward-compatible migrations.
Phase 3: Data Quality & Observability
- Implement data quality checks at every pipeline stage: freshness, completeness, uniqueness, referential integrity.
- Build data lineage graphs tracking field-level transformations from source to dashboard.
- Configure pipeline monitoring: job duration, row counts, failure rates, SLA breach alerts.
Verification & Quality Checklist
- [ ] All pipelines are idempotent and safely re-runnable without data duplication.
- [ ] Schema changes are backward compatible and documented in migration logs.
- [ ] Data quality tests cover freshness (data is recent), volume (expected row counts), and validity (no nulls in required fields).
- [ ] Pipeline SLAs defined with automated alerting on breach.
Anti-Patterns & Constraints
- NEVER build pipelines without idempotency guarantees.
- NEVER perform destructive schema changes (column drops/renames) without a migration period.
- NEVER skip data quality validation between ingestion and serving layers.
04 同仓库更多 Skills
workflow-optimizer
Moved to workflow-mapping in skills-agents. Removed in v4.1.
webhooks
Build webhook receivers with signature verification, idempotency, retries and dead letter queues. Use when implementing, verifying, or signing incoming/outgoing webhooks.
webhook-handler-builder
Moved to webhooks in skills-developer. Removed in v4.1.
voice-ai-integration-engineer
Moved to speech-pipelines in skills-developer. Removed in v4.1.
typescript-migrator
Moved to port-code in skills-developer. Removed in v4.1.
trigger-dev
Moved to trigger-dev in skills-agents. Removed in v4.1.