[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M792DRM2E613RMZPVJ8NE47Q":3,"skill-same:01M792DRM2E613RMZPVJ8NE47Q":61},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":26,"commitSha":27,"skillDirectory":28,"forksCount":14,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":32,"ownerDescription":33,"updatedAt":34,"install":35,"resources":48},"01M792DRM2E613RMZPVJ8NE47Q","data-pipelines",null,"Build ETL\u002FELT and streaming pipelines with quality validation, schema evolution and anomaly correction. Use when building ETL\u002FELT pipelines, streaming data, or schema validation.","poorvith-mp\u002Fskills-developer","poorvith-mp","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F277478855?v=4","01M78Z5K0YH1Z3PX815PGQ96ND","Poorvith M P","01M792ACSFXT45RJ69SSGKTVJN",0,"MIT","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fdata-pipelines\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: data-pipelines\nlast_reviewed: 2026-09-06\ngroup: Data and AI\ndescription: >-\n  Build ETL\u002FELT and streaming pipelines with quality validation, schema evolution and anomaly\n  correction. Use when building ETL\u002FELT pipelines, streaming data, or schema validation.\n---\n\n# Data Pipelines\n\nArchitects reliable, scalable data pipelines from ingestion through transformation to serving, with built-in quality validation and schema governance.\n\n## Phased Workflow\n\n### Phase 1: Source Analysis & Architecture Selection\n1. Inventory data sources: APIs, databases, event streams, file drops, SaaS webhooks.\n2. Classify each pipeline: Batch (hourly\u002Fdaily, high volume) vs Streaming (real-time, event-driven).\n3. Select architecture pattern: ELT (extract-load-transform in warehouse) vs ETL (transform before load).\n\n### Phase 2: Pipeline Implementation\n1. Design idempotent, retry-safe extraction with checkpoint\u002Foffset tracking.\n2. Implement transformations using dbt models (staging → intermediate → marts) or Spark jobs.\n3. Define schema evolution strategy: additive-only columns, versioned schemas, backward-compatible migrations.\n\n### Phase 3: Data Quality & Observability\n1. Implement data quality checks at every pipeline stage: freshness, completeness, uniqueness, referential integrity.\n2. Build data lineage graphs tracking field-level transformations from source to dashboard.\n3. Configure pipeline monitoring: job duration, row counts, failure rates, SLA breach alerts.\n\n## Verification & Quality Checklist\n- [ ] All pipelines are idempotent and safely re-runnable without data duplication.\n- [ ] Schema changes are backward compatible and documented in migration logs.\n- [ ] Data quality tests cover freshness (data is recent), volume (expected row counts), and validity (no nulls in required fields).\n- [ ] Pipeline SLAs defined with automated alerting on breach.\n\n## Anti-Patterns & Constraints\n- NEVER build pipelines without idempotency guarantees.\n- NEVER perform destructive schema changes (column drops\u002Frenames) without a migration period.\n- NEVER skip data quality validation between ingestion and serving layers.\n","# Data Pipelines\n\nArchitects reliable, scalable data pipelines from ingestion through transformation to serving, with built-in quality validation and schema governance.\n\n## Phased Workflow\n\n### Phase 1: Source Analysis & Architecture Selection\n1. Inventory data sources: APIs, databases, event streams, file drops, SaaS webhooks.\n2. Classify each pipeline: Batch (hourly\u002Fdaily, high volume) vs Streaming (real-time, event-driven).\n3. Select architecture pattern: ELT (extract-load-transform in warehouse) vs ETL (transform before load).\n\n### Phase 2: Pipeline Implementation\n1. Design idempotent, retry-safe extraction with checkpoint\u002Foffset tracking.\n2. Implement transformations using dbt models (staging → intermediate → marts) or Spark jobs.\n3. Define schema evolution strategy: additive-only columns, versioned schemas, backward-compatible migrations.\n\n### Phase 3: Data Quality & Observability\n1. Implement data quality checks at every pipeline stage: freshness, completeness, uniqueness, referential integrity.\n2. Build data lineage graphs tracking field-level transformations from source to dashboard.\n3. Configure pipeline monitoring: job duration, row counts, failure rates, SLA breach alerts.\n\n## Verification & Quality Checklist\n- [ ] All pipelines are idempotent and safely re-runnable without data duplication.\n- [ ] Schema changes are backward compatible and documented in migration logs.\n- [ ] Data quality tests cover freshness (data is recent), volume (expected row counts), and validity (no nulls in required fields).\n- [ ] Pipeline SLAs defined with automated alerting on breach.\n\n## Anti-Patterns & Constraints\n- NEVER build pipelines without idempotency guarantees.\n- NEVER perform destructive schema changes (column drops\u002Frenames) without a migration period.\n- NEVER skip data quality validation between ingestion and serving layers.\n",{"name":5,"group":24,"description":7,"last_reviewed":25},"Data and AI",1788652800000,"main","f6a053c46d4bf0f779ea766c929b6f6f92c51765","skills\u002Fdata-pipelines","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskillary","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp","poorvithmp.com","Shipping tools for problems people notice too late.","2026-10-02 21:32:22",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add poorvith-mp\u002Fskills-developer --skill data-pipelines",{"tool":41,"command":42},"bunx","bunx skills add poorvith-mp\u002Fskills-developer --skill data-pipelines",{"tool":44,"command":45},"pnpm","pnpm dlx skills add poorvith-mp\u002Fskills-developer --skill data-pipelines","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fdata-pipelines\u002FSKILL.md",true,[49,55],{"path":50,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":53,"sourceHtmlUrl":54},"data-pipelines.skill","OTHER","其他",1815,"https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fdata-pipelines\u002Fdata-pipelines.skill",{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"evals\u002Frouting.yaml","REFERENCE","参考文档",864,"https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fdata-pipelines\u002Fevals\u002Frouting.yaml",[62,69,76,83,90,97],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M792Q79X14YD94WXEEJY9GCW","workflow-optimizer","Moved to workflow-mapping in skills-agents. Removed in v4.1.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fworkflow-optimizer\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M792Q387DNC91VKEVM07SJM3","webhooks","Build webhook receivers with signature verification, idempotency, retries and dead letter queues. Use when implementing, verifying, or signing incoming\u002Foutgoing webhooks.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fwebhooks\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M792PZW0SM71X7V9709ZWHCR","webhook-handler-builder","Moved to webhooks in skills-developer. Removed in v4.1.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fwebhook-handler-builder\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M792PW7XPB7J9S0QSS52JXZV","voice-ai-integration-engineer","Moved to speech-pipelines in skills-developer. Removed in v4.1.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Fvoice-ai-integration-engineer\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M792PRZ5GSAQ1P2999V9XAJ5","typescript-migrator","Moved to port-code in skills-developer. Removed in v4.1.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Ftypescript-migrator\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M792PQDARKP00V8Z3GYD4QZP","trigger-dev","Moved to trigger-dev in skills-agents. Removed in v4.1.","https:\u002F\u002Fgithub.com\u002Fpoorvith-mp\u002Fskills-developer\u002Fblob\u002Fmain\u002Fskills\u002Ftrigger-dev\u002FSKILL.md",[],[]]