skilltta-task-guidance
Temporary guidance for implementing the Flink LongestSessionPerJob task over Google cluster-usage traces
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- skilltta-task-guidance
- description
- Temporary guidance for implementing the Flink LongestSessionPerJob task over Google cluster-usage traces
03 SKILL.md 正文
Task Guidance: LongestSessionPerJob (Flink)
Problem Restatement
For each finished job, identify "stages" from its task SUBMIT events. A stage is a run of SUBMIT events separated by < 10 minutes of inactivity (in event time). Report the task count of the job's largest stage.
Critical Correctness Rules (derive from the spec, not assumptions)
Rule A — Output qualification: only FINISHED jobs
The spec says "once the job has finished". Do not emit a tuple for every jobId that appears in task events. You must:
- Read the job-events stream and detect job completion (Google traces job event types: 0=SUBMIT, 1=SCHEDULE, 2=EVICT, 3=FAIL, 4=FINISH, 5=KILL, 6=LOST; treat FINISH/FAIL/KILL per the format doc — confirm by reading
format.pdf/ClusterData2011_2.md; the spec wording "finished" typically means event type 4 FINISH). - Only output
(jobId, maxStageTaskCount)for jobs whose job-event stream contains a terminal/finish event. - A job that has task events but never finishes in the input window must not appear in the output.
Implementation sketch: keep per-job stage state keyed by jobId from the task stream, and gate emission on a signal from the job stream (e.g., connect the two streams via connect + KeyedCoProcessFunction, or union with a tagged type).
Rule B — Only SUBMIT task events count
Filter task events to event type SUBMIT (type 0) before stage logic. Resubmits (SUBMIT after FAIL/EVICT) are counted as new tasks within whatever stage they land in — do not dedupe by taskIndex.
Rule C — Timestamps are microseconds
- 10 minutes =
10 * 60 * 1_000_000Lmicroseconds. - Assign event-time timestamps and watermarks in microseconds (don't silently convert to ms).
Rule D — Stage boundary = 10-min gap of SUBMITs for that job
Use event-time session windowing per jobId with a 10-minute gap, counting records per window. Then reduce to the max count per jobId. Options:
keyBy(jobId).window(EventTimeSessionWindows.withGap(Time.minutes(10)))with a count aggregator, then keyBy jobId again and keep max.- Or manual
KeyedProcessFunctionwith a timer atlastSubmitTs + 10minthat flushes the current stage count into a running max.
Output Format
- One line per finished job:
(jobId,maxCount)— literal parentheses, comma separated, no spaces. - Order does not matter.
- Write to the local path given by
outputparameter. UseStreamingFileSinkwith a single-file sink or (simpler for batch-style CSV input) collect results and write viaPrintSink/writeAsTexttooutputwith parallelism 1 to avoid directory output.
Required Data Classes (under clusterdata.datatypes)
Implement at minimum:
TaskEventwith fields:timestamp(long, μs),jobId(long),taskIndex(long/int),eventType(int), plus any required byAppBase.JobEventwith fields:timestamp(long, μs),jobId(long),eventType(int). Provide no-arg constructors + public fields or getters (POJO) so Flink serializes efficiently.
Parsing Pattern (transferable from CSV examples)
From the retrieved CSV-processing patterns:
- Treat each gzipped CSV as newline-delimited rows; split by
,; index into fields performat.pdf. - Guard against empty/short lines (skip).
- Parse ints/longs with try/catch; drop malformed rows (don't crash the job).
- Reading
.gz: useTextInputFormaton the.gzpath (Flink auto-decompresses), orenv.readTextFile(path).
Suggested Pipeline
taskStream = env.readTextFile(task_input).map(parseTaskEvent).filter(eventType==SUBMIT).assignTimestampsAndWatermarks(...μs...).jobFinishStream = env.readTextFile(job_input).map(parseJobEvent).filter(isTerminal).assignTimestampsAndWatermarks(...).- Session-window task stream by jobId with 10-min gap →
(jobId, stageCount). - KeyBy jobId → reduce to max stage count per job (keyed state).
- Connect with
jobFinishStreamkeyed by jobId; on finish signal, emit(jobId, currentMax)and clear state. - Sink with parallelism 1 to
outputpath, formatting as(jobId,count).
Common Pitfalls to Avoid
- ❌ Emitting a line per jobId seen in task events (fails Rule A — the location hint flags exactly this).
- ❌ Using processing time or Time.minutes assumed to be ms when ts is μs.
- ❌ Counting non-SUBMIT task events.
- ❌ Deduping resubmits of the same taskIndex.
- ❌ Writing to
outputas a directory via defaultwriteAsTextwith parallelism>1. - ❌ Changing the class name
LongestSessionPerJobor editingpom.xml.
Verification Before Finishing
- Jar builds via provided
pom.xml. - Running with the sample paths produces a file at
outputwhere every line matches^\(\d+,\d+\)$. - Every jobId in output has a terminal event in
job_input; jobIds without one are absent.
04 同仓库更多 Skills
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms