[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BK00GYY7ECD52Q3WF6GC75":3,"skill-same:01M7BK00GYY7ECD52Q3WF6GC75":160},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":85,"commitSha":86,"skillDirectory":87,"forksCount":88,"repositoryHtmlUrl":89,"repositoryHomepageUrl":6,"ownerHtmlUrl":90,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":91,"install":92,"resources":104},"01M7BK00GYY7ECD52Q3WF6GC75","senior-data-engineer",null,"World-class data engineering skill for building scalable data pipelines, ETL\u002FELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.","linyouxue\u002Fskill-repair-benchmark","linyouxue","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F216151464?v=4","01M7BJEVCXKD74KH6FG9CBVSQB","林宥学","01M7BJEVE02NP217V93BKZFC0J",0,"MIT","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\n\n# === CORE IDENTITY ===\nname: senior-data-engineer\ntitle: Senior Data Engineer Skill Package\ndescription: World-class data engineering skill for building scalable data pipelines, ETL\u002FELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.\ndomain: engineering\nsubdomain: data-engineering\n\n# === WEBSITE DISPLAY ===\ndifficulty: advanced\ntime-saved: \"TODO: Quantify time savings\"\nfrequency: \"TODO: Estimate usage frequency\"\nuse-cases:\n  - Designing data pipelines for ETL\u002FELT processes\n  - Building data warehouses and data lakes\n  - Implementing data quality and governance frameworks\n  - Creating analytics dashboards and reporting\n  - Building real-time streaming pipelines with Kafka and Flink\n  - Implementing exactly-once streaming semantics\n  - Monitoring streaming quality (consumer lag, data freshness, schema drift)\n\n# === RELATIONSHIPS ===\nrelated-agents: []\nrelated-skills: []\nrelated-commands: []\norchestrated-by: []\n\n# === TECHNICAL ===\ndependencies:\n  scripts: []\n  references: []\n  assets: []\ncompatibility: \"Python 3.8+; platforms: macos, linux, windows\"\ntech-stack:\n  - Python\n  - SQL\n  - Apache Spark\n  - Airflow\n  - dbt\n  - Apache Kafka\n  - Apache Flink\n  - AWS Kinesis\n  - Spark Structured Streaming\n  - Kafka Streams\n  - PostgreSQL\n  - BigQuery\n  - Snowflake\n  - Docker\n  - Schema Registry\n\n# === EXAMPLES ===\nexamples:\n  -\n    title: Example Usage\n    input: \"TODO: Add example input for senior-data-engineer\"\n    output: \"TODO: Add expected output\"\n\n# === ANALYTICS ===\nstats:\n  downloads: 0\n  stars: 0\n  rating: 0.0\n  reviews: 0\n\n# === VERSIONING ===\nversion: v2.0.0\nauthor: Claude Skills Team\ncontributors: []\ncreated: 2025-10-20\nupdated: 2025-12-16\nlicense: MIT\n\n# === DISCOVERABILITY ===\ntags: [architecture, data, design, engineer, engineering, senior, streaming, kafka, flink, real-time]\nfeatured: false\nverified: true\n---\n\n# Senior Data Engineer\n\n## Core Capabilities\n\n- **Batch Pipeline Orchestration** - Design and implement production-ready ETL\u002FELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring\n- **Real-Time Streaming** - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency\n- **Data Quality Management** - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity\n- **Streaming Quality Monitoring** - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines\n- **Performance Optimization** - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations\n\n\n## Key Workflows\n\n### Workflow 1: Build ETL Pipeline\n\n**Time:** 2-4 hours\n\n**Steps:**\n1. Design pipeline architecture using Lambda, Kappa, or Medallion pattern\n2. Configure YAML pipeline definition with sources, transformations, targets\n3. Generate Airflow DAG with `pipeline_orchestrator.py`\n4. Define data quality validation rules\n5. Deploy and configure monitoring\u002Falerting\n\n**Expected Output:** Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring\n\n### Workflow 2: Build Real-Time Streaming Pipeline\n\n**Time:** 3-5 days\n\n**Steps:**\n1. Select streaming architecture (Kappa vs Lambda) based on requirements\n2. Confirm schema + time semantics from available docs and a few raw records (e.g., column counts, **timestamp units like ms vs µs**, timestamp ordering\u002Flate-data expectations); ensure your input layer can actually read the raw format end-to-end (e.g., **gzipped CSV**), then configure streaming pipeline YAML (sources, processing, sinks, quality)\n3. Generate Kafka configurations with `kafka_config_generator.py`\n4. Use the provided job skeleton or generate Flink\u002FSpark job scaffolding with `stream_processor.py`; implement the actual parsing + state\u002Fwindow logic **and the result emission rule** (continuous vs “emit once when complete”). Run a clean local build early (`mvn -DskipTests package` or equivalent) plus a small end-to-end smoke test (and, when semantics are tricky, an independent cross-check script over a small sample). If Maven dependency\u002Fplugin fetch fails in the environment (e.g., HTTP 403 from Central), fix the build configuration first (HTTPS mirror \u002F provided repo) before iterating on logic. When the requirement is a **single local output file**, set sink parallelism to `1` and emit the exact line format required by the consumer\u002Ftests.\n5. Deploy and monitor with `streaming_quality_validator.py`\n\n**Expected Output:** Streaming pipeline processing 10K+ events\u002Fsec with P99 latency \u003C 1s, exactly-once delivery, and real-time quality monitoring\n\n\nWorld-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.\n\n## Overview\n\nThis skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL\u002FELT design, data quality frameworks, pipeline orchestration, and DataOps practices.\n\n**What This Skill Provides:**\n- Production-ready pipeline templates (Airflow, Spark, dbt)\n- Comprehensive data quality validation framework\n- Performance optimization and cost analysis tools\n- Data architecture patterns (Lambda, Kappa, Medallion)\n- Complete DataOps CI\u002FCD workflows\n\n**Best For:**\n- Building scalable data pipelines for enterprise systems\n- Implementing data quality and governance frameworks\n- Optimizing ETL performance and cloud costs\n- Designing modern data architectures (lake, warehouse, lakehouse)\n- Production ML\u002FAI data infrastructure\n\n## Quick Start\n\n### Pipeline Orchestration\n\n```bash\n# Generate Airflow DAG from configuration\npython scripts\u002Fpipeline_orchestrator.py --config pipeline_config.yaml --output dags\u002F\n\n# Validate pipeline configuration\npython scripts\u002Fpipeline_orchestrator.py --config pipeline_config.yaml --validate\n\n# Use incremental load template\npython scripts\u002Fpipeline_orchestrator.py --template incremental --output dags\u002F\n```\n\n### Data Quality Validation\n\n```bash\n# Validate CSV file with quality checks\npython scripts\u002Fdata_quality_validator.py --input data\u002Fsales.csv --output report.html\n\n# Validate database table with custom rules\npython scripts\u002Fdata_quality_validator.py \\\n    --connection postgresql:\u002F\u002Fuser:pass@host\u002Fdb \\\n    --table sales_transactions \\\n    --rules rules\u002Fsales_validation.yaml \\\n    --threshold 0.95\n```\n\n### Performance Optimization\n\n```bash\n# Analyze pipeline performance and get recommendations\npython scripts\u002Fetl_performance_optimizer.py \\\n    --airflow-db postgresql:\u002F\u002Fhost\u002Fairflow \\\n    --dag-id sales_etl_pipeline \\\n    --days 30 \\\n    --optimize\n\n# Analyze Spark job performance\npython scripts\u002Fetl_performance_optimizer.py \\\n    --spark-history-server http:\u002F\u002Fspark-history:18080 \\\n    --app-id app-20250115-001\n```\n\n### Real-Time Streaming\n\n```bash\n# Validate streaming pipeline configuration\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --validate\n\n# Generate Kafka topic and client configurations\npython scripts\u002Fkafka_config_generator.py \\\n    --topic user-events \\\n    --partitions 12 \\\n    --replication 3 \\\n    --output kafka\u002Ftopics\u002F\n\n# Generate exactly-once producer configuration\npython scripts\u002Fkafka_config_generator.py \\\n    --producer \\\n    --profile exactly-once \\\n    --output kafka\u002Fproducer.properties\n\n# Generate Flink job scaffolding\npython scripts\u002Fstream_processor.py \\\n    --config streaming_config.yaml \\\n    --mode flink \\\n    --generate \\\n    --output flink-jobs\u002F\n\n# Monitor streaming quality\npython scripts\u002Fstreaming_quality_validator.py \\\n    --lag --consumer-group events-processor --threshold 10000 \\\n    --freshness --topic processed-events --max-latency-ms 5000 \\\n    --output streaming-health-report.html\n```\n\n## Core Workflows\n\n### 1. Building Production Data Pipelines\n\n**Steps:**\n1. **Design Architecture:** Choose pattern (Lambda, Kappa, Medallion) based on requirements\n2. **Configure Pipeline:** Create YAML configuration with sources, transformations, targets\n3. **Generate DAG:** `python scripts\u002Fpipeline_orchestrator.py --config config.yaml`\n4. **Add Quality Checks:** Define validation rules for data quality\n5. **Deploy & Monitor:** Deploy to Airflow, configure alerts, track metrics\n\n**Pipeline Patterns:** See [frameworks.md](references\u002Fframeworks.md) for Lambda Architecture, Kappa Architecture, Medallion Architecture (Bronze\u002FSilver\u002FGold), and Microservices Data patterns.\n\n**Templates:** See [templates.md](references\u002Ftemplates.md) for complete Airflow DAG templates, Spark job templates, dbt models, and Docker configurations.\n\n### 2. Data Quality Management\n\n**Steps:**\n1. **Define Rules:** Create validation rules covering completeness, accuracy, consistency\n2. **Run Validation:** `python scripts\u002Fdata_quality_validator.py --rules rules.yaml`\n3. **Review Results:** Analyze quality scores and failed checks\n4. **Integrate CI\u002FCD:** Add validation to pipeline deployment process\n5. **Monitor Trends:** Track quality scores over time\n\n**Quality Framework:** See [frameworks.md](references\u002Fframeworks.md) for complete Data Quality Framework covering all dimensions (completeness, accuracy, consistency, timeliness, validity).\n\n**Validation Templates:** See [templates.md](references\u002Ftemplates.md) for validation configuration examples and Python API usage.\n\n### 3. Data Modeling & Transformation\n\n**Steps:**\n1. **Choose Modeling Approach:** Dimensional (Kimball), Data Vault 2.0, or One Big Table\n2. **Design Schema:** Define fact tables, dimensions, and relationships\n3. **Implement with dbt:** Create staging, intermediate, and mart models\n4. **Handle SCD:** Implement slowly changing dimension logic (Type 1\u002F2\u002F3)\n5. **Test & Deploy:** Run dbt tests, generate documentation, deploy\n\n**Modeling Patterns:** See [frameworks.md](references\u002Fframeworks.md) for Dimensional Modeling (Kimball), Data Vault 2.0, One Big Table (OBT), and SCD implementations.\n\n**dbt Templates:** See [templates.md](references\u002Ftemplates.md) for complete dbt model templates including staging, intermediate, fact tables, and SCD Type 2 logic.\n\n### 4. Performance Optimization\n\n**Steps:**\n1. **Profile Pipeline:** Run performance analyzer on recent pipeline executions\n2. **Identify Bottlenecks:** Review execution time breakdown and slow tasks\n3. **Apply Optimizations:** Implement recommendations (partitioning, indexing, batching)\n4. **Tune Spark Jobs:** Optimize memory, parallelism, and shuffle settings\n5. **Measure Impact:** Compare before\u002Fafter metrics, track cost savings\n\n**Optimization Strategies:** See [frameworks.md](references\u002Fframeworks.md) for performance best practices including partitioning strategies, query optimization, and Spark tuning.\n\n**Analysis Tools:** See [tools.md](references\u002Ftools.md) for complete documentation on etl_performance_optimizer.py with query analysis and Spark tuning.\n\n### 5. Building Real-Time Streaming Pipelines\n\n**Steps:**\n1. **Architecture Selection:** Choose Kappa (streaming-only) or Lambda (batch + streaming) architecture\n2. **Configure Pipeline:** Create YAML config with sources, processing engine, sinks, quality thresholds\n3. **Generate Kafka Configs:** `python scripts\u002Fkafka_config_generator.py --topic events --partitions 12`\n4. **Generate Job Scaffolding:** `python scripts\u002Fstream_processor.py --mode flink --generate`\n5. **Deploy Infrastructure:** Use Docker Compose for local dev, Kubernetes for production\n6. **Monitor Quality:** `python scripts\u002Fstreaming_quality_validator.py --lag --freshness --throughput`\n\n**Streaming Patterns:** See [frameworks.md](references\u002Fframeworks.md) for stateful processing, stream joins, windowing, exactly-once semantics, and CDC patterns.\n\n**Templates:** See [templates.md](references\u002Ftemplates.md) for Flink DataStream jobs, Kafka Streams applications, PyFlink templates, and Docker Compose configurations.\n\n#### Semantic Contract Gate for Keyed Event Aggregations\n\nBefore implementing a keyed aggregation that combines activity events with lifecycle events, write down and preserve three separate predicates:\n\n1. **Contributing-event predicate:** which event types add observations to a window, stage, or aggregate.\n2. **Completion predicate:** the event condition that authorizes final output (a specific label when the spec names one, or a defined set of terminal labels when the spec uses a generic concept like “finished\u002Fterminated”).\n3. **Emission\u002Fflush predicate:** the mechanism that guarantees you’ve seen all contributing events that should count before you output (e.g., emit immediately on the completion event when ordering is guaranteed; otherwise wait for the watermark to pass the completion time using a **watermark-safe timer** such as `completionTs + 1`, and ensure bounded inputs advance watermarks to completion so timers actually fire).\n\nTreat event labels as distinct states unless the specification groups them. If a specification says to emit after a specific event (e.g., `FINISH`), require that label. If the specification only says the entity is “finished\u002Fterminated” (without naming labels), define the completion predicate as the full set of terminal labels present in the lifecycle stream and verify coverage by scanning counts per terminal label (so you don’t silently drop keys that end via a different terminal label).\n\nBefore declaring the job complete, validate semantics with small hand-checkable fixtures:\n\n- a key with qualifying activity and a completion event (per your completion predicate) must be emitted;\n- a key with qualifying activity but no completion event must not be emitted;\n- if the completion predicate is a set, include fixtures that exercise multiple terminal labels from the set and confirm they all produce output; compare the output key set as well as aggregate values, and investigate unexpected extra keys even when all expected keys are present;\n- include at least one fixture where the completion event is at the end of the input (to catch “timer never fired \u002F watermark never advanced” output bugs).\n\n## Python Tools\n\n### pipeline_orchestrator.py\n\nAutomated Airflow DAG generation with intelligent dependency resolution and monitoring.\n\n**Key Features:**\n- Generate production-ready DAGs from YAML configuration\n- Automatic task dependency resolution\n- Built-in retry logic and error handling\n- Multi-source support (PostgreSQL, S3, BigQuery, Snowflake)\n- Integrated quality checks and alerting\n\n**Usage:**\n```bash\n# Basic DAG generation\npython scripts\u002Fpipeline_orchestrator.py --config pipeline_config.yaml --output dags\u002F\n\n# With validation\npython scripts\u002Fpipeline_orchestrator.py --config config.yaml --validate\n\n# From template\npython scripts\u002Fpipeline_orchestrator.py --template incremental --output dags\u002F\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for full configuration options, templates, and integration examples.\n\n### data_quality_validator.py\n\nComprehensive data quality validation framework with automated checks and reporting.\n\n**Capabilities:**\n- Multi-dimensional validation (completeness, accuracy, consistency, timeliness, validity)\n- Great Expectations integration\n- Custom business rule validation\n- HTML\u002FPDF report generation\n- Anomaly detection\n- Historical trend tracking\n\n**Usage:**\n```bash\n# Validate with custom rules\npython scripts\u002Fdata_quality_validator.py \\\n    --input data\u002Fsales.csv \\\n    --rules rules\u002Fsales_validation.yaml \\\n    --output report.html\n\n# Database table validation\npython scripts\u002Fdata_quality_validator.py \\\n    --connection postgresql:\u002F\u002Fhost\u002Fdb \\\n    --table sales_transactions \\\n    --threshold 0.95\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for rule configuration, API usage, and integration patterns.\n\n### etl_performance_optimizer.py\n\nPipeline performance analysis with actionable optimization recommendations.\n\n**Capabilities:**\n- Airflow DAG execution profiling\n- Bottleneck detection and analysis\n- SQL query optimization suggestions\n- Spark job tuning recommendations\n- Cost analysis and optimization\n- Historical performance trending\n\n**Usage:**\n```bash\n# Analyze Airflow DAG\npython scripts\u002Fetl_performance_optimizer.py \\\n    --airflow-db postgresql:\u002F\u002Fhost\u002Fairflow \\\n    --dag-id sales_etl_pipeline \\\n    --days 30 \\\n    --optimize\n\n# Spark job analysis\npython scripts\u002Fetl_performance_optimizer.py \\\n    --spark-history-server http:\u002F\u002Fspark-history:18080 \\\n    --app-id app-20250115-001\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for profiling options, optimization strategies, and cost analysis.\n\n### stream_processor.py\n\nStreaming pipeline configuration generator and validator for Kafka, Flink, and Kinesis.\n\n**Capabilities:**\n- Multi-platform support (Kafka, Flink, Kinesis, Spark Streaming)\n- Configuration validation with best practice checks\n- Flink\u002FSpark job scaffolding generation\n- Kafka topic configuration generation\n- Docker Compose for local streaming stacks\n- Exactly-once semantics configuration\n\n**Usage:**\n```bash\n# Validate configuration\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --validate\n\n# Generate Kafka configurations\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --mode kafka --generate\n\n# Generate Flink job scaffolding\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --mode flink --generate --output flink-jobs\u002F\n\n# Generate Docker Compose for local development\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --mode docker --generate\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for configuration format, validation checks, and generated outputs.\n\n### streaming_quality_validator.py\n\nReal-time streaming data quality monitoring with comprehensive health scoring.\n\n**Capabilities:**\n- Consumer lag monitoring with thresholds\n- Data freshness validation (P50\u002FP95\u002FP99 latency)\n- Schema drift detection\n- Throughput analysis (events\u002Fsec, bytes\u002Fsec)\n- Dead letter queue rate monitoring\n- Overall quality scoring with recommendations\n- Prometheus metrics export\n\n**Usage:**\n```bash\n# Monitor consumer lag\npython scripts\u002Fstreaming_quality_validator.py \\\n    --lag --consumer-group events-processor --threshold 10000\n\n# Monitor data freshness\npython scripts\u002Fstreaming_quality_validator.py \\\n    --freshness --topic processed-events --max-latency-ms 5000\n\n# Full quality validation\npython scripts\u002Fstreaming_quality_validator.py \\\n    --lag --freshness --throughput --dlq \\\n    --output streaming-health-report.html\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for all monitoring dimensions and integration patterns.\n\n### kafka_config_generator.py\n\nProduction-grade Kafka configuration generator with performance and security profiles.\n\n**Capabilities:**\n- Topic configuration (partitions, replication, retention, compaction)\n- Producer profiles (high-throughput, exactly-once, low-latency, ordered)\n- Consumer profiles (exactly-once, high-throughput, batch)\n- Kafka Streams configuration with state store tuning\n- Security configuration (SASL-PLAIN, SASL-SCRAM, mTLS)\n- Kafka Connect source\u002Fsink configurations\n- Multiple output formats (properties, YAML, JSON)\n\n**Usage:**\n```bash\n# Generate topic configuration\npython scripts\u002Fkafka_config_generator.py \\\n    --topic user-events --partitions 12 --replication 3 --retention-hours 168\n\n# Generate exactly-once producer\npython scripts\u002Fkafka_config_generator.py \\\n    --producer --profile exactly-once --transactional-id producer-001\n\n# Generate Kafka Streams config\npython scripts\u002Fkafka_config_generator.py \\\n    --streams --application-id events-processor --exactly-once\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for all profiles, security options, and Connect configurations.\n\n## Reference Documentation\n\n### Frameworks ([frameworks.md](references\u002Fframeworks.md))\n\nComprehensive data engineering frameworks and patterns:\n- **Architecture Patterns:** Lambda, Kappa, Medallion, Microservices data architecture\n- **Data Modeling:** Dimensional (Kimball), Data Vault 2.0, One Big Table\n- **ETL\u002FELT Patterns:** Full load, incremental load, CDC, SCD, idempotent pipelines\n- **Data Quality:** Complete framework covering all quality dimensions\n- **DataOps:** CI\u002FCD for data pipelines, testing strategies, monitoring\n- **Orchestration:** Airflow DAG patterns, backfill strategies\n- **Real-Time Streaming:** Stateful processing, stream joins, windowing strategies, exactly-once semantics, event time processing, watermarks, backpressure, Apache Flink patterns, AWS Kinesis patterns, CDC for streaming\n- **Governance:** Data catalog, lineage tracking, access control\n\n### Templates ([templates.md](references\u002Ftemplates.md))\n\nProduction-ready code templates and examples:\n- **Airflow DAGs:** Complete ETL DAG, incremental load, dynamic task generation\n- **Spark Jobs:** Batch processing, streaming, optimized configurations\n- **dbt Models:** Staging, intermediate, fact tables, dimensions with SCD Type 2\n- **SQL Patterns:** Incremental merge (upsert), deduplication, date spine, window functions\n- **Python Pipelines:** Data quality validation class, retry decorators, error handling\n- **Real-Time Streaming:** Apache Flink DataStream jobs (Java), Kafka Streams applications, PyFlink jobs, AWS Kinesis consumers, Docker Compose for streaming stack\n- **Kafka Configs:** Producer\u002Fconsumer properties templates, topic configurations, security configurations\n- **Docker:** Dockerfiles for data pipelines, Docker Compose for local development including streaming stack (Kafka, Flink, Schema Registry)\n- **Configuration:** dbt project config, Spark configuration, Airflow variables, streaming pipeline YAML\n- **Testing:** pytest fixtures, integration tests, data quality tests\n\n### Tools ([tools.md](references\u002Ftools.md))\n\nPython automation tool documentation:\n- **pipeline_orchestrator.py:** Complete usage guide, configuration format, DAG templates\n- **data_quality_validator.py:** Validation rules, dimension checks, Great Expectations integration\n- **etl_performance_optimizer.py:** Performance analysis, query optimization, Spark tuning\n- **stream_processor.py:** Streaming pipeline configuration, validation, job scaffolding generation\n- **streaming_quality_validator.py:** Consumer lag, data freshness, schema drift, throughput monitoring\n- **kafka_config_generator.py:** Topic, producer, consumer, Kafka Streams, and Connect configurations\n- **Integration Patterns:** Airflow, dbt, CI\u002FCD, monitoring systems, Prometheus\n- **Best Practices:** Configuration management, error handling, performance, monitoring, streaming quality\n\n## Tech Stack\n\n**Core Technologies:**\n- **Languages:** Python 3.8+, SQL, Scala (Spark), Java (Flink)\n- **Orchestration:** Apache Airflow, Prefect, Dagster\n- **Batch Processing:** Apache Spark, dbt, Pandas\n- **Stream Processing:** Apache Kafka, Apache Flink, Kafka Streams, Spark Structured Streaming, AWS Kinesis\n- **Storage:** PostgreSQL, BigQuery, Snowflake, Redshift, S3, GCS\n- **Schema Management:** Confluent Schema Registry, AWS Glue Schema Registry\n- **Containerization:** Docker, Kubernetes\n- **Monitoring:** Datadog, Prometheus, Grafana, Kafka UI\n\n**Data Platforms:**\n- **Cloud Data Warehouses:** Snowflake, BigQuery, Redshift\n- **Data Lakes:** Delta Lake, Apache Iceberg, Apache Hudi\n- **Streaming Platforms:** Apache Kafka, AWS Kinesis, Google Pub\u002FSub, Azure Event Hubs\n- **Stream Processing Engines:** Apache Flink, Kafka Streams, Spark Structured Streaming\n- **Workflow:** Airflow, Prefect, Dagster\n\n## Integration Points\n\nThis skill integrates with:\n- **Orchestration:** Airflow, Prefect, Dagster for workflow management\n- **Transformation:** dbt for SQL transformations and testing\n- **Quality:** Great Expectations for data validation\n- **Monitoring:** Datadog, Prometheus for pipeline monitoring\n- **BI Tools:** Looker, Tableau, Power BI for analytics\n- **ML Platforms:** MLflow, Kubeflow for ML pipeline integration\n- **Version Control:** Git for pipeline code and configuration\n\nSee [tools.md](references\u002Ftools.md) for detailed integration patterns and examples.\n\n## Best Practices\n\n**Pipeline Design:**\n1. Idempotent operations for safe reruns\n2. Incremental processing where possible\n3. Clear data lineage and documentation\n4. Comprehensive error handling\n5. Automated recovery mechanisms\n\n**Data Quality:**\n1. Define quality rules early\n2. Validate at every pipeline stage\n3. Automate quality monitoring\n4. Track quality trends over time\n5. Block bad data from downstream\n\n**Performance:**\n1. Partition large tables by date\u002Fregion\n2. Use columnar formats (Parquet, ORC)\n3. Leverage predicate pushdown\n4. Optimize for your query patterns\n5. Monitor and tune regularly\n\n**Operations:**\n1. Version control everything\n2. Automate testing and deployment\n3. Implement comprehensive monitoring\n4. Document runbooks for incidents\n5. Regular performance reviews\n\n## Performance Targets\n\n**Batch Pipeline Execution:**\n- P50 latency: \u003C 5 minutes (hourly pipelines)\n- P95 latency: \u003C 15 minutes\n- Success rate: > 99%\n- Data freshness: \u003C 1 hour behind source\n\n**Streaming Pipeline Execution:**\n- Throughput: 10K+ events\u002Fsecond sustained\n- End-to-end latency: P99 \u003C 1 second\n- Consumer lag: \u003C 10K records behind\n- Exactly-once delivery: Zero duplicates or losses\n\n**Data Quality (Batch):**\n- Quality score: > 95%\n- Completeness: > 99%\n- Timeliness: \u003C 2 hours data lag\n- Zero critical failures\n\n**Streaming Quality:**\n- Data freshness: P95 \u003C 5 minutes from event generation\n- Late data rate: \u003C 5% outside watermark window\n- Dead letter queue rate: \u003C 1%\n- Schema compatibility: 100% backward\u002Fforward compatible changes\n\n**Cost Efficiency:**\n- Cost per GB processed: \u003C $0.10\n- Cloud cost trend: Stable or decreasing\n- Resource utilization: > 70%\n\n## Resources\n\n- **Frameworks Guide:** [references\u002Fframeworks.md](references\u002Fframeworks.md)\n- **Code Templates:** [references\u002Ftemplates.md](references\u002Ftemplates.md)\n- **Tool Documentation:** [references\u002Ftools.md](references\u002Ftools.md)\n- **Python Scripts:** `scripts\u002F` directory\n\n---\n\n**Version:** 2.0.0\n**Last Updated:** December 16, 2025\n**Documentation Structure:** Progressive disclosure with comprehensive references\n**Streaming Enhancement:** Task #8 - Real-time streaming capabilities added\n","# Senior Data Engineer\n\n## Core Capabilities\n\n- **Batch Pipeline Orchestration** - Design and implement production-ready ETL\u002FELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring\n- **Real-Time Streaming** - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency\n- **Data Quality Management** - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity\n- **Streaming Quality Monitoring** - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines\n- **Performance Optimization** - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations\n\n\n## Key Workflows\n\n### Workflow 1: Build ETL Pipeline\n\n**Time:** 2-4 hours\n\n**Steps:**\n1. Design pipeline architecture using Lambda, Kappa, or Medallion pattern\n2. Configure YAML pipeline definition with sources, transformations, targets\n3. Generate Airflow DAG with `pipeline_orchestrator.py`\n4. Define data quality validation rules\n5. Deploy and configure monitoring\u002Falerting\n\n**Expected Output:** Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring\n\n### Workflow 2: Build Real-Time Streaming Pipeline\n\n**Time:** 3-5 days\n\n**Steps:**\n1. Select streaming architecture (Kappa vs Lambda) based on requirements\n2. Confirm schema + time semantics from available docs and a few raw records (e.g., column counts, **timestamp units like ms vs µs**, timestamp ordering\u002Flate-data expectations); ensure your input layer can actually read the raw format end-to-end (e.g., **gzipped CSV**), then configure streaming pipeline YAML (sources, processing, sinks, quality)\n3. Generate Kafka configurations with `kafka_config_generator.py`\n4. Use the provided job skeleton or generate Flink\u002FSpark job scaffolding with `stream_processor.py`; implement the actual parsing + state\u002Fwindow logic **and the result emission rule** (continuous vs “emit once when complete”). Run a clean local build early (`mvn -DskipTests package` or equivalent) plus a small end-to-end smoke test (and, when semantics are tricky, an independent cross-check script over a small sample). If Maven dependency\u002Fplugin fetch fails in the environment (e.g., HTTP 403 from Central), fix the build configuration first (HTTPS mirror \u002F provided repo) before iterating on logic. When the requirement is a **single local output file**, set sink parallelism to `1` and emit the exact line format required by the consumer\u002Ftests.\n5. Deploy and monitor with `streaming_quality_validator.py`\n\n**Expected Output:** Streaming pipeline processing 10K+ events\u002Fsec with P99 latency \u003C 1s, exactly-once delivery, and real-time quality monitoring\n\n\nWorld-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.\n\n## Overview\n\nThis skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL\u002FELT design, data quality frameworks, pipeline orchestration, and DataOps practices.\n\n**What This Skill Provides:**\n- Production-ready pipeline templates (Airflow, Spark, dbt)\n- Comprehensive data quality validation framework\n- Performance optimization and cost analysis tools\n- Data architecture patterns (Lambda, Kappa, Medallion)\n- Complete DataOps CI\u002FCD workflows\n\n**Best For:**\n- Building scalable data pipelines for enterprise systems\n- Implementing data quality and governance frameworks\n- Optimizing ETL performance and cloud costs\n- Designing modern data architectures (lake, warehouse, lakehouse)\n- Production ML\u002FAI data infrastructure\n\n## Quick Start\n\n### Pipeline Orchestration\n\n```bash\n# Generate Airflow DAG from configuration\npython scripts\u002Fpipeline_orchestrator.py --config pipeline_config.yaml --output dags\u002F\n\n# Validate pipeline configuration\npython scripts\u002Fpipeline_orchestrator.py --config pipeline_config.yaml --validate\n\n# Use incremental load template\npython scripts\u002Fpipeline_orchestrator.py --template incremental --output dags\u002F\n```\n\n### Data Quality Validation\n\n```bash\n# Validate CSV file with quality checks\npython scripts\u002Fdata_quality_validator.py --input data\u002Fsales.csv --output report.html\n\n# Validate database table with custom rules\npython scripts\u002Fdata_quality_validator.py \\\n    --connection postgresql:\u002F\u002Fuser:pass@host\u002Fdb \\\n    --table sales_transactions \\\n    --rules rules\u002Fsales_validation.yaml \\\n    --threshold 0.95\n```\n\n### Performance Optimization\n\n```bash\n# Analyze pipeline performance and get recommendations\npython scripts\u002Fetl_performance_optimizer.py \\\n    --airflow-db postgresql:\u002F\u002Fhost\u002Fairflow \\\n    --dag-id sales_etl_pipeline \\\n    --days 30 \\\n    --optimize\n\n# Analyze Spark job performance\npython scripts\u002Fetl_performance_optimizer.py \\\n    --spark-history-server http:\u002F\u002Fspark-history:18080 \\\n    --app-id app-20250115-001\n```\n\n### Real-Time Streaming\n\n```bash\n# Validate streaming pipeline configuration\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --validate\n\n# Generate Kafka topic and client configurations\npython scripts\u002Fkafka_config_generator.py \\\n    --topic user-events \\\n    --partitions 12 \\\n    --replication 3 \\\n    --output kafka\u002Ftopics\u002F\n\n# Generate exactly-once producer configuration\npython scripts\u002Fkafka_config_generator.py \\\n    --producer \\\n    --profile exactly-once \\\n    --output kafka\u002Fproducer.properties\n\n# Generate Flink job scaffolding\npython scripts\u002Fstream_processor.py \\\n    --config streaming_config.yaml \\\n    --mode flink \\\n    --generate \\\n    --output flink-jobs\u002F\n\n# Monitor streaming quality\npython scripts\u002Fstreaming_quality_validator.py \\\n    --lag --consumer-group events-processor --threshold 10000 \\\n    --freshness --topic processed-events --max-latency-ms 5000 \\\n    --output streaming-health-report.html\n```\n\n## Core Workflows\n\n### 1. Building Production Data Pipelines\n\n**Steps:**\n1. **Design Architecture:** Choose pattern (Lambda, Kappa, Medallion) based on requirements\n2. **Configure Pipeline:** Create YAML configuration with sources, transformations, targets\n3. **Generate DAG:** `python scripts\u002Fpipeline_orchestrator.py --config config.yaml`\n4. **Add Quality Checks:** Define validation rules for data quality\n5. **Deploy & Monitor:** Deploy to Airflow, configure alerts, track metrics\n\n**Pipeline Patterns:** See [frameworks.md](references\u002Fframeworks.md) for Lambda Architecture, Kappa Architecture, Medallion Architecture (Bronze\u002FSilver\u002FGold), and Microservices Data patterns.\n\n**Templates:** See [templates.md](references\u002Ftemplates.md) for complete Airflow DAG templates, Spark job templates, dbt models, and Docker configurations.\n\n### 2. Data Quality Management\n\n**Steps:**\n1. **Define Rules:** Create validation rules covering completeness, accuracy, consistency\n2. **Run Validation:** `python scripts\u002Fdata_quality_validator.py --rules rules.yaml`\n3. **Review Results:** Analyze quality scores and failed checks\n4. **Integrate CI\u002FCD:** Add validation to pipeline deployment process\n5. **Monitor Trends:** Track quality scores over time\n\n**Quality Framework:** See [frameworks.md](references\u002Fframeworks.md) for complete Data Quality Framework covering all dimensions (completeness, accuracy, consistency, timeliness, validity).\n\n**Validation Templates:** See [templates.md](references\u002Ftemplates.md) for validation configuration examples and Python API usage.\n\n### 3. Data Modeling & Transformation\n\n**Steps:**\n1. **Choose Modeling Approach:** Dimensional (Kimball), Data Vault 2.0, or One Big Table\n2. **Design Schema:** Define fact tables, dimensions, and relationships\n3. **Implement with dbt:** Create staging, intermediate, and mart models\n4. **Handle SCD:** Implement slowly changing dimension logic (Type 1\u002F2\u002F3)\n5. **Test & Deploy:** Run dbt tests, generate documentation, deploy\n\n**Modeling Patterns:** See [frameworks.md](references\u002Fframeworks.md) for Dimensional Modeling (Kimball), Data Vault 2.0, One Big Table (OBT), and SCD implementations.\n\n**dbt Templates:** See [templates.md](references\u002Ftemplates.md) for complete dbt model templates including staging, intermediate, fact tables, and SCD Type 2 logic.\n\n### 4. Performance Optimization\n\n**Steps:**\n1. **Profile Pipeline:** Run performance analyzer on recent pipeline executions\n2. **Identify Bottlenecks:** Review execution time breakdown and slow tasks\n3. **Apply Optimizations:** Implement recommendations (partitioning, indexing, batching)\n4. **Tune Spark Jobs:** Optimize memory, parallelism, and shuffle settings\n5. **Measure Impact:** Compare before\u002Fafter metrics, track cost savings\n\n**Optimization Strategies:** See [frameworks.md](references\u002Fframeworks.md) for performance best practices including partitioning strategies, query optimization, and Spark tuning.\n\n**Analysis Tools:** See [tools.md](references\u002Ftools.md) for complete documentation on etl_performance_optimizer.py with query analysis and Spark tuning.\n\n### 5. Building Real-Time Streaming Pipelines\n\n**Steps:**\n1. **Architecture Selection:** Choose Kappa (streaming-only) or Lambda (batch + streaming) architecture\n2. **Configure Pipeline:** Create YAML config with sources, processing engine, sinks, quality thresholds\n3. **Generate Kafka Configs:** `python scripts\u002Fkafka_config_generator.py --topic events --partitions 12`\n4. **Generate Job Scaffolding:** `python scripts\u002Fstream_processor.py --mode flink --generate`\n5. **Deploy Infrastructure:** Use Docker Compose for local dev, Kubernetes for production\n6. **Monitor Quality:** `python scripts\u002Fstreaming_quality_validator.py --lag --freshness --throughput`\n\n**Streaming Patterns:** See [frameworks.md](references\u002Fframeworks.md) for stateful processing, stream joins, windowing, exactly-once semantics, and CDC patterns.\n\n**Templates:** See [templates.md](references\u002Ftemplates.md) for Flink DataStream jobs, Kafka Streams applications, PyFlink templates, and Docker Compose configurations.\n\n#### Semantic Contract Gate for Keyed Event Aggregations\n\nBefore implementing a keyed aggregation that combines activity events with lifecycle events, write down and preserve three separate predicates:\n\n1. **Contributing-event predicate:** which event types add observations to a window, stage, or aggregate.\n2. **Completion predicate:** the event condition that authorizes final output (a specific label when the spec names one, or a defined set of terminal labels when the spec uses a generic concept like “finished\u002Fterminated”).\n3. **Emission\u002Fflush predicate:** the mechanism that guarantees you’ve seen all contributing events that should count before you output (e.g., emit immediately on the completion event when ordering is guaranteed; otherwise wait for the watermark to pass the completion time using a **watermark-safe timer** such as `completionTs + 1`, and ensure bounded inputs advance watermarks to completion so timers actually fire).\n\nTreat event labels as distinct states unless the specification groups them. If a specification says to emit after a specific event (e.g., `FINISH`), require that label. If the specification only says the entity is “finished\u002Fterminated” (without naming labels), define the completion predicate as the full set of terminal labels present in the lifecycle stream and verify coverage by scanning counts per terminal label (so you don’t silently drop keys that end via a different terminal label).\n\nBefore declaring the job complete, validate semantics with small hand-checkable fixtures:\n\n- a key with qualifying activity and a completion event (per your completion predicate) must be emitted;\n- a key with qualifying activity but no completion event must not be emitted;\n- if the completion predicate is a set, include fixtures that exercise multiple terminal labels from the set and confirm they all produce output; compare the output key set as well as aggregate values, and investigate unexpected extra keys even when all expected keys are present;\n- include at least one fixture where the completion event is at the end of the input (to catch “timer never fired \u002F watermark never advanced” output bugs).\n\n## Python Tools\n\n### pipeline_orchestrator.py\n\nAutomated Airflow DAG generation with intelligent dependency resolution and monitoring.\n\n**Key Features:**\n- Generate production-ready DAGs from YAML configuration\n- Automatic task dependency resolution\n- Built-in retry logic and error handling\n- Multi-source support (PostgreSQL, S3, BigQuery, Snowflake)\n- Integrated quality checks and alerting\n\n**Usage:**\n```bash\n# Basic DAG generation\npython scripts\u002Fpipeline_orchestrator.py --config pipeline_config.yaml --output dags\u002F\n\n# With validation\npython scripts\u002Fpipeline_orchestrator.py --config config.yaml --validate\n\n# From template\npython scripts\u002Fpipeline_orchestrator.py --template incremental --output dags\u002F\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for full configuration options, templates, and integration examples.\n\n### data_quality_validator.py\n\nComprehensive data quality validation framework with automated checks and reporting.\n\n**Capabilities:**\n- Multi-dimensional validation (completeness, accuracy, consistency, timeliness, validity)\n- Great Expectations integration\n- Custom business rule validation\n- HTML\u002FPDF report generation\n- Anomaly detection\n- Historical trend tracking\n\n**Usage:**\n```bash\n# Validate with custom rules\npython scripts\u002Fdata_quality_validator.py \\\n    --input data\u002Fsales.csv \\\n    --rules rules\u002Fsales_validation.yaml \\\n    --output report.html\n\n# Database table validation\npython scripts\u002Fdata_quality_validator.py \\\n    --connection postgresql:\u002F\u002Fhost\u002Fdb \\\n    --table sales_transactions \\\n    --threshold 0.95\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for rule configuration, API usage, and integration patterns.\n\n### etl_performance_optimizer.py\n\nPipeline performance analysis with actionable optimization recommendations.\n\n**Capabilities:**\n- Airflow DAG execution profiling\n- Bottleneck detection and analysis\n- SQL query optimization suggestions\n- Spark job tuning recommendations\n- Cost analysis and optimization\n- Historical performance trending\n\n**Usage:**\n```bash\n# Analyze Airflow DAG\npython scripts\u002Fetl_performance_optimizer.py \\\n    --airflow-db postgresql:\u002F\u002Fhost\u002Fairflow \\\n    --dag-id sales_etl_pipeline \\\n    --days 30 \\\n    --optimize\n\n# Spark job analysis\npython scripts\u002Fetl_performance_optimizer.py \\\n    --spark-history-server http:\u002F\u002Fspark-history:18080 \\\n    --app-id app-20250115-001\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for profiling options, optimization strategies, and cost analysis.\n\n### stream_processor.py\n\nStreaming pipeline configuration generator and validator for Kafka, Flink, and Kinesis.\n\n**Capabilities:**\n- Multi-platform support (Kafka, Flink, Kinesis, Spark Streaming)\n- Configuration validation with best practice checks\n- Flink\u002FSpark job scaffolding generation\n- Kafka topic configuration generation\n- Docker Compose for local streaming stacks\n- Exactly-once semantics configuration\n\n**Usage:**\n```bash\n# Validate configuration\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --validate\n\n# Generate Kafka configurations\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --mode kafka --generate\n\n# Generate Flink job scaffolding\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --mode flink --generate --output flink-jobs\u002F\n\n# Generate Docker Compose for local development\npython scripts\u002Fstream_processor.py --config streaming_config.yaml --mode docker --generate\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for configuration format, validation checks, and generated outputs.\n\n### streaming_quality_validator.py\n\nReal-time streaming data quality monitoring with comprehensive health scoring.\n\n**Capabilities:**\n- Consumer lag monitoring with thresholds\n- Data freshness validation (P50\u002FP95\u002FP99 latency)\n- Schema drift detection\n- Throughput analysis (events\u002Fsec, bytes\u002Fsec)\n- Dead letter queue rate monitoring\n- Overall quality scoring with recommendations\n- Prometheus metrics export\n\n**Usage:**\n```bash\n# Monitor consumer lag\npython scripts\u002Fstreaming_quality_validator.py \\\n    --lag --consumer-group events-processor --threshold 10000\n\n# Monitor data freshness\npython scripts\u002Fstreaming_quality_validator.py \\\n    --freshness --topic processed-events --max-latency-ms 5000\n\n# Full quality validation\npython scripts\u002Fstreaming_quality_validator.py \\\n    --lag --freshness --throughput --dlq \\\n    --output streaming-health-report.html\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for all monitoring dimensions and integration patterns.\n\n### kafka_config_generator.py\n\nProduction-grade Kafka configuration generator with performance and security profiles.\n\n**Capabilities:**\n- Topic configuration (partitions, replication, retention, compaction)\n- Producer profiles (high-throughput, exactly-once, low-latency, ordered)\n- Consumer profiles (exactly-once, high-throughput, batch)\n- Kafka Streams configuration with state store tuning\n- Security configuration (SASL-PLAIN, SASL-SCRAM, mTLS)\n- Kafka Connect source\u002Fsink configurations\n- Multiple output formats (properties, YAML, JSON)\n\n**Usage:**\n```bash\n# Generate topic configuration\npython scripts\u002Fkafka_config_generator.py \\\n    --topic user-events --partitions 12 --replication 3 --retention-hours 168\n\n# Generate exactly-once producer\npython scripts\u002Fkafka_config_generator.py \\\n    --producer --profile exactly-once --transactional-id producer-001\n\n# Generate Kafka Streams config\npython scripts\u002Fkafka_config_generator.py \\\n    --streams --application-id events-processor --exactly-once\n```\n\n**Complete Documentation:** See [tools.md](references\u002Ftools.md) for all profiles, security options, and Connect configurations.\n\n## Reference Documentation\n\n### Frameworks ([frameworks.md](references\u002Fframeworks.md))\n\nComprehensive data engineering frameworks and patterns:\n- **Architecture Patterns:** Lambda, Kappa, Medallion, Microservices data architecture\n- **Data Modeling:** Dimensional (Kimball), Data Vault 2.0, One Big Table\n- **ETL\u002FELT Patterns:** Full load, incremental load, CDC, SCD, idempotent pipelines\n- **Data Quality:** Complete framework covering all quality dimensions\n- **DataOps:** CI\u002FCD for data pipelines, testing strategies, monitoring\n- **Orchestration:** Airflow DAG patterns, backfill strategies\n- **Real-Time Streaming:** Stateful processing, stream joins, windowing strategies, exactly-once semantics, event time processing, watermarks, backpressure, Apache Flink patterns, AWS Kinesis patterns, CDC for streaming\n- **Governance:** Data catalog, lineage tracking, access control\n\n### Templates ([templates.md](references\u002Ftemplates.md))\n\nProduction-ready code templates and examples:\n- **Airflow DAGs:** Complete ETL DAG, incremental load, dynamic task generation\n- **Spark Jobs:** Batch processing, streaming, optimized configurations\n- **dbt Models:** Staging, intermediate, fact tables, dimensions with SCD Type 2\n- **SQL Patterns:** Incremental merge (upsert), deduplication, date spine, window functions\n- **Python Pipelines:** Data quality validation class, retry decorators, error handling\n- **Real-Time Streaming:** Apache Flink DataStream jobs (Java), Kafka Streams applications, PyFlink jobs, AWS Kinesis consumers, Docker Compose for streaming stack\n- **Kafka Configs:** Producer\u002Fconsumer properties templates, topic configurations, security configurations\n- **Docker:** Dockerfiles for data pipelines, Docker Compose for local development including streaming stack (Kafka, Flink, Schema Registry)\n- **Configuration:** dbt project config, Spark configuration, Airflow variables, streaming pipeline YAML\n- **Testing:** pytest fixtures, integration tests, data quality tests\n\n### Tools ([tools.md](references\u002Ftools.md))\n\nPython automation tool documentation:\n- **pipeline_orchestrator.py:** Complete usage guide, configuration format, DAG templates\n- **data_quality_validator.py:** Validation rules, dimension checks, Great Expectations integration\n- **etl_performance_optimizer.py:** Performance analysis, query optimization, Spark tuning\n- **stream_processor.py:** Streaming pipeline configuration, validation, job scaffolding generation\n- **streaming_quality_validator.py:** Consumer lag, data freshness, schema drift, throughput monitoring\n- **kafka_config_generator.py:** Topic, producer, consumer, Kafka Streams, and Connect configurations\n- **Integration Patterns:** Airflow, dbt, CI\u002FCD, monitoring systems, Prometheus\n- **Best Practices:** Configuration management, error handling, performance, monitoring, streaming quality\n\n## Tech Stack\n\n**Core Technologies:**\n- **Languages:** Python 3.8+, SQL, Scala (Spark), Java (Flink)\n- **Orchestration:** Apache Airflow, Prefect, Dagster\n- **Batch Processing:** Apache Spark, dbt, Pandas\n- **Stream Processing:** Apache Kafka, Apache Flink, Kafka Streams, Spark Structured Streaming, AWS Kinesis\n- **Storage:** PostgreSQL, BigQuery, Snowflake, Redshift, S3, GCS\n- **Schema Management:** Confluent Schema Registry, AWS Glue Schema Registry\n- **Containerization:** Docker, Kubernetes\n- **Monitoring:** Datadog, Prometheus, Grafana, Kafka UI\n\n**Data Platforms:**\n- **Cloud Data Warehouses:** Snowflake, BigQuery, Redshift\n- **Data Lakes:** Delta Lake, Apache Iceberg, Apache Hudi\n- **Streaming Platforms:** Apache Kafka, AWS Kinesis, Google Pub\u002FSub, Azure Event Hubs\n- **Stream Processing Engines:** Apache Flink, Kafka Streams, Spark Structured Streaming\n- **Workflow:** Airflow, Prefect, Dagster\n\n## Integration Points\n\nThis skill integrates with:\n- **Orchestration:** Airflow, Prefect, Dagster for workflow management\n- **Transformation:** dbt for SQL transformations and testing\n- **Quality:** Great Expectations for data validation\n- **Monitoring:** Datadog, Prometheus for pipeline monitoring\n- **BI Tools:** Looker, Tableau, Power BI for analytics\n- **ML Platforms:** MLflow, Kubeflow for ML pipeline integration\n- **Version Control:** Git for pipeline code and configuration\n\nSee [tools.md](references\u002Ftools.md) for detailed integration patterns and examples.\n\n## Best Practices\n\n**Pipeline Design:**\n1. Idempotent operations for safe reruns\n2. Incremental processing where possible\n3. Clear data lineage and documentation\n4. Comprehensive error handling\n5. Automated recovery mechanisms\n\n**Data Quality:**\n1. Define quality rules early\n2. Validate at every pipeline stage\n3. Automate quality monitoring\n4. Track quality trends over time\n5. Block bad data from downstream\n\n**Performance:**\n1. Partition large tables by date\u002Fregion\n2. Use columnar formats (Parquet, ORC)\n3. Leverage predicate pushdown\n4. Optimize for your query patterns\n5. Monitor and tune regularly\n\n**Operations:**\n1. Version control everything\n2. Automate testing and deployment\n3. Implement comprehensive monitoring\n4. Document runbooks for incidents\n5. Regular performance reviews\n\n## Performance Targets\n\n**Batch Pipeline Execution:**\n- P50 latency: \u003C 5 minutes (hourly pipelines)\n- P95 latency: \u003C 15 minutes\n- Success rate: > 99%\n- Data freshness: \u003C 1 hour behind source\n\n**Streaming Pipeline Execution:**\n- Throughput: 10K+ events\u002Fsecond sustained\n- End-to-end latency: P99 \u003C 1 second\n- Consumer lag: \u003C 10K records behind\n- Exactly-once delivery: Zero duplicates or losses\n\n**Data Quality (Batch):**\n- Quality score: > 95%\n- Completeness: > 99%\n- Timeliness: \u003C 2 hours data lag\n- Zero critical failures\n\n**Streaming Quality:**\n- Data freshness: P95 \u003C 5 minutes from event generation\n- Late data rate: \u003C 5% outside watermark window\n- Dead letter queue rate: \u003C 1%\n- Schema compatibility: 100% backward\u002Fforward compatible changes\n\n**Cost Efficiency:**\n- Cost per GB processed: \u003C $0.10\n- Cloud cost trend: Stable or decreasing\n- Resource utilization: > 70%\n\n## Resources\n\n- **Frameworks Guide:** [references\u002Fframeworks.md](references\u002Fframeworks.md)\n- **Code Templates:** [references\u002Ftemplates.md](references\u002Ftemplates.md)\n- **Tool Documentation:** [references\u002Ftools.md](references\u002Ftools.md)\n- **Python Scripts:** `scripts\u002F` directory\n\n---\n\n**Version:** 2.0.0\n**Last Updated:** December 16, 2025\n**Documentation Structure:** Progressive disclosure with comprehensive references\n**Streaming Enhancement:** Task #8 - Real-time streaming capabilities added\n",{"name":5,"tags":24,"stats":35,"title":36,"author":37,"domain":29,"created":38,"license":15,"updated":39,"version":40,"examples":41,"featured":20,"verified":46,"frequency":47,"subdomain":48,"use-cases":49,"difficulty":57,"tech-stack":58,"time-saved":74,"description":7,"contributors":75,"dependencies":76,"compatibility":80,"related-agents":81,"related-skills":82,"orchestrated-by":83,"related-commands":84},[25,26,27,28,29,30,31,32,33,34],"architecture","data","design","engineer","engineering","senior","streaming","kafka","flink","real-time",{"stars":14,"rating":14,"reviews":14,"downloads":14},"Senior Data Engineer Skill Package","Claude Skills Team",1760918400000,1765843200000,"v2.0.0",[42],{"input":43,"title":44,"output":45},"TODO: Add example input for senior-data-engineer","Example Usage","TODO: Add expected output",true,"TODO: Estimate usage frequency","data-engineering",[50,51,52,53,54,55,56],"Designing data pipelines for ETL\u002FELT processes","Building data warehouses and data lakes","Implementing data quality and governance frameworks","Creating analytics dashboards and reporting","Building real-time streaming pipelines with Kafka and Flink","Implementing exactly-once streaming semantics","Monitoring streaming quality (consumer lag, data freshness, schema drift)","advanced",[59,60,61,62,63,64,65,66,67,68,69,70,71,72,73],"Python","SQL","Apache Spark","Airflow","dbt","Apache Kafka","Apache Flink","AWS Kinesis","Spark Structured Streaming","Kafka Streams","PostgreSQL","BigQuery","Snowflake","Docker","Schema Registry","TODO: Quantify time savings",[],{"assets":77,"scripts":78,"references":79},[],[],[],"Python 3.8+; platforms: macos, linux, windows",[],[],[],[],"main","cdcb6d230f3b15c3c33f8a2bdd16439f43602cbe","controlled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer",1,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark","https:\u002F\u002Fgithub.com\u002Flinyouxue","2026-11-14 20:28:35",{"source":8,"skillName":5,"commands":93,"prompt":103,"zipAvailable":46},[94,97,100],{"tool":95,"command":96},"npx","npx skills add linyouxue\u002Fskill-repair-benchmark --skill senior-data-engineer",{"tool":98,"command":99},"bunx","bunx skills add linyouxue\u002Fskill-repair-benchmark --skill senior-data-engineer",{"tool":101,"command":102},"pnpm","pnpm dlx skills add linyouxue\u002Fskill-repair-benchmark --skill senior-data-engineer","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002FSKILL.md",[105,110,113,116,122,126,130,134,140,144,148,152,156],{"path":106,"resourceType":107,"resourceTypeDesc":108,"sizeBytes":14,"sourceHtmlUrl":109},"assets\u002F.gitkeep","ASSET","资产文件","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fassets\u002F.gitkeep",{"path":111,"resourceType":107,"resourceTypeDesc":108,"sizeBytes":14,"sourceHtmlUrl":112},"assets\u002Fdata-quality-checklist.md","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fassets\u002Fdata-quality-checklist.md",{"path":114,"resourceType":107,"resourceTypeDesc":108,"sizeBytes":14,"sourceHtmlUrl":115},"assets\u002Fpipeline-design-template.md","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fassets\u002Fpipeline-design-template.md",{"path":117,"resourceType":118,"resourceTypeDesc":119,"sizeBytes":120,"sourceHtmlUrl":121},"HOW_TO_USE.md","REFERENCE","参考文档",2500,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002FHOW_TO_USE.md",{"path":123,"resourceType":118,"resourceTypeDesc":119,"sizeBytes":124,"sourceHtmlUrl":125},"references\u002Fframeworks.md",42313,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Freferences\u002Fframeworks.md",{"path":127,"resourceType":118,"resourceTypeDesc":119,"sizeBytes":128,"sourceHtmlUrl":129},"references\u002Ftemplates.md",67136,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Freferences\u002Ftemplates.md",{"path":131,"resourceType":118,"resourceTypeDesc":119,"sizeBytes":132,"sourceHtmlUrl":133},"references\u002Ftools.md",39140,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Freferences\u002Ftools.md",{"path":135,"resourceType":136,"resourceTypeDesc":137,"sizeBytes":138,"sourceHtmlUrl":139},"scripts\u002Fdata_quality_validator.py","SCRIPT","脚本",3489,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fscripts\u002Fdata_quality_validator.py",{"path":141,"resourceType":136,"resourceTypeDesc":137,"sizeBytes":142,"sourceHtmlUrl":143},"scripts\u002Fetl_performance_optimizer.py",3504,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fscripts\u002Fetl_performance_optimizer.py",{"path":145,"resourceType":136,"resourceTypeDesc":137,"sizeBytes":146,"sourceHtmlUrl":147},"scripts\u002Fkafka_config_generator.py",41177,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fscripts\u002Fkafka_config_generator.py",{"path":149,"resourceType":136,"resourceTypeDesc":137,"sizeBytes":150,"sourceHtmlUrl":151},"scripts\u002Fpipeline_orchestrator.py",3487,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fscripts\u002Fpipeline_orchestrator.py",{"path":153,"resourceType":136,"resourceTypeDesc":137,"sizeBytes":154,"sourceHtmlUrl":155},"scripts\u002Fstream_processor.py",60553,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fscripts\u002Fstream_processor.py",{"path":157,"resourceType":136,"resourceTypeDesc":137,"sizeBytes":158,"sourceHtmlUrl":159},"scripts\u002Fstreaming_quality_validator.py",40005,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fcontrolled-skill-repair-experiments\u002Fresult\u002Fmmg2skill-gpt5.2-xhigh\u002Fcandidates\u002Fsingle-flink-query-RI-002\u002Fround-3\u002Fskills\u002Fsenior-data-engineer\u002Fscripts\u002Fstreaming_quality_validator.py",[161,169,176,183,188,193],{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":165,"licenseFileUrl":6,"sourceHtmlUrl":166,"categories":167,"keywords":168,"publishedAt":19},"01M7E1HGVRY73NBJ022CJG13X6","video-frame-extraction","Extract frames from video files and save them as images using OpenCV","Apache-2.0","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[],[],{"skillNo":170,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":165,"licenseFileUrl":6,"sourceHtmlUrl":173,"categories":174,"keywords":175,"publishedAt":19},"01M7E1HAG5R7CJ9C8NYJS771MB","openai-vision","Analyze images and multi-frame sequences using OpenAI GPT vision models","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[],{"skillNo":177,"name":178,"nameZh":6,"description":179,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":165,"licenseFileUrl":6,"sourceHtmlUrl":180,"categories":181,"keywords":182,"publishedAt":19},"01M7E1H7DADKB6QBWKC2KQ63SZ","image-ocr","Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[],[],{"skillNo":184,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":165,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":187,"publishedAt":19},"01M7E1H0WBNT43TENGQY7AVZHD","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[],[],{"skillNo":189,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":165,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":192,"publishedAt":19},"01M7E1GTC7D4N4Q2EF03875J9E","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[],{"skillNo":194,"name":178,"nameZh":6,"description":179,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":165,"licenseFileUrl":6,"sourceHtmlUrl":195,"categories":196,"keywords":197,"publishedAt":19},"01M7E1GQ92CWFMGH53KSSE4WER","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[],[]]