senior-data-engineer
Implement and debug a Flink DataStream job that sessionizes task SUBMIT events per job (10-minute inactivity gap), computes the maximum tasks in any stage, and emits results once the job is finished.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- senior-data-engineer
- tags
- [ "architecture", "data", "design", "engineer", "engineering", "senior", "streaming", "kafka", "flink", "real-time" ]
- stats
- { "stars": 0, "rating": 0, "reviews": 0, "downloads": 0 }
- title
- Senior Data Engineer Skill Package
- author
- Claude Skills Team
- domain
- engineering
- created
- 1760918400000
- license
- MIT
- updated
- 1765843200000
- version
- v2.0.0
- examples
- [ { "input": "TODO: Add example input for senior-data-engineer", "title": "Example Usage", "output": "TODO: Add expected output" } ]
- featured
- false
- verified
- true
- frequency
- TODO: Estimate usage frequency
- subdomain
- data-engineering
- use-cases
- [ "Designing data pipelines for ETL/ELT processes", "Building data warehouses and data lakes", "Implementing data quality and governance frameworks", "Creating analytics dashboards and reporting", "Building real-time streaming pipelines with Kafka and Flink", "Implementing exactly-once streaming semantics", "Monitoring streaming quality (consumer lag, data freshness, schema drift)" ]
- difficulty
- advanced
- tech-stack
- [ "Python", "SQL", "Apache Spark", "Airflow", "dbt", "Apache Kafka", "Apache Flink", "AWS Kinesis", "Spark Structured Streaming", "Kafka Streams", "PostgreSQL", "BigQuery", "Snowflake", "Docker", "Schema Registry" ]
- time-saved
- TODO: Quantify time savings
- description
- Implement and debug a Flink DataStream job that sessionizes task SUBMIT events per job (10-minute inactivity gap), computes the maximum tasks in any stage, and emits results once the job is finished.
- contributors
- []
- dependencies
- { "assets": [], "scripts": [], "references": [] }
- compatibility
- Python 3.8+; platforms: macos, linux, windows
- related-agents
- []
- related-skills
- []
- orchestrated-by
- []
- related-commands
- []
02 资源文件 13
- 资产文件assets/.gitkeep—
- 资产文件assets/data-quality-checklist.md—
- 资产文件assets/pipeline-design-template.md—
- 参考文档HOW_TO_USE.md2.4 KB
- 参考文档references/frameworks.md41.3 KB
- 参考文档references/templates.md65.6 KB
- 参考文档references/tools.md38.2 KB
- 脚本scripts/data_quality_validator.py3.4 KB
- 脚本scripts/etl_performance_optimizer.py3.4 KB
- 脚本scripts/kafka_config_generator.py40.2 KB
- 脚本scripts/pipeline_orchestrator.py3.4 KB
- 脚本scripts/stream_processor.py59.1 KB
- 脚本scripts/streaming_quality_validator.py39.1 KB
03 SKILL.md 正文
Steps
- Read schema docs (
format.pdf,ClusterData2011_2.md) and implement the required datatypes (e.g.,TaskEvent,JobEvent) underclusterdata.datatypesmatching the CSV column order and microsecond timestamps. - In the Flink job:
- Ingest a single gzipped CSV shard for task events and job events.
- Parse events and assign event-time timestamps in microseconds; use a watermark strategy appropriate for the input (bounded files).
- Filter and stage task submissions:
- Keep only task
SUBMITevents. keyBy(jobId)and build event-time session “stages” with a 10-minute inactivity gap (10 * 60 * 1_000_000microseconds).- Count the number of SUBMIT events per session window.
- Reduce per
jobIdto the maximum session count (longest stage).
- Keep only task
- Emit only when the job is done:
- From job events, derive a “terminal” signal per
jobId(include at leastFINISH, and also treatKILLas terminal when present). - Join / connect the terminal signal with the computed max-stage count so output is produced “once the job has finished” (jobs with no matching submits in the chosen task shard may legitimately produce
0).
- From job events, derive a “terminal” signal per
- Prevent Flink generic type-erasure runtime failures:
- If using generic
SourceFunction/ parsing operators (e.g.,GzipCsvSourceFunction<T>), add explicit type hints on the resultingDataStream(e.g.,.returns(TaskEvent.class)/.returns(JobEvent.class)) at the point where Flink can’t inferT.
- If using generic
- Write results to the provided
outputpath:- One tuple per line in exactly:
(jobId,longestStageTaskCount) - Ordering not required.
- One tuple per line in exactly:
Expected Result
The job compiles and runs on Flink without InvalidTypesException from generics, and writes a local output file containing one line per jobId with the longest stage’s SUBMIT-count, emitting results only after terminal job completion (including killed jobs).
04 同仓库更多 Skills
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms