skilltta-task-guidance
Guidance for implementing the Flink LongestSessionPerJob task over Google cluster-usage traces, with corrections for stage/session detection and output eligibility.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- skilltta-task-guidance
- description
- Guidance for implementing the Flink LongestSessionPerJob task over Google cluster-usage traces, with corrections for stage/session detection and output eligibility.
03 SKILL.md 正文
SkillTTA Task Guidance: Longest Stage Per Finished Job (Flink)
Context
You are implementing clusterdata.query.LongestSessionPerJob (Flink streaming job) that:
- Reads a gzipped task_events CSV and a gzipped job_events CSV (Google cluster-usage traces, timestamps in microseconds).
- Groups task SUBMIT events (event type = 0) per
jobIdinto "stages" separated by >=10 minutes of inactivity (session gap on event time). - For each finished job, emits
(jobId, maxTasksInAnyStage)to a local output file, one tuple per line.
Data schema reminders
- task_events columns:
timestamp, missing_info, job_id, task_index, machine_id, event_type, user, scheduling_class, priority, cpu, memory, disk, different_machine. - job_events columns:
timestamp, missing_info, job_id, event_type, user, scheduling_class, job_name, logical_job_name. - Event types (both tables): 0=SUBMIT, 1=SCHEDULE, 2=EVICT, 3=FAIL, 4=FINISH, 5=KILL, 6=LOST, 7=UPDATE_PENDING, 8=UPDATE_RUNNING.
- Timestamps are microseconds since trace start. 10 minutes =
10 * 60 * 1_000_000microseconds.
Workflow 1 — Project wiring
- Keep class name
LongestSessionPerJoband extendclusterdata.utils.AppBase(do not editpom.xml). - Create POJO/Tuple classes under
clusterdata.datatypes, e.g.TaskEventandJobEvent, withtimestamp (long µs),jobId (long),taskIndex (long, task only),eventType (int), and other fields you need. Provide no-arg constructor + public fields (or getters/setters) so Flink's POJO serializer works. - Parse CSVs with a
GZIPInputStreamreader source orreadTextFile+ manual gunzip; emit events with an event-timeWatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(0))keyed bytimestamp(convert µs → ms for Flink watermarks via/1000). Make sure to assign timestamps in microseconds consistently inside your own state logic.
Workflow 2 — Stage detection and job-finish gating (CRITICAL)
This is where prior attempts commonly fail; follow these rules exactly.
2.1 What counts as a task SUBMIT in a stage
- Only task events with
event_type == 0(SUBMIT) contribute to stage task counts. - If a task is submitted, then FAILS/EVICTED/KILLED and later re-SUBMITTED, each SUBMIT is counted separately (do not deduplicate by
(jobId, taskIndex)).
2.2 Session/stage gap rule
- Use event time gaps, not processing time.
- A stage ends when there is no SUBMIT for that
jobIdfor >= 10 minutes (600_000_000 µs). - Implement with either:
- Flink
EventTimeSessionWindows.withGap(Time.minutes(10))keyed byjobIdover the SUBMIT stream, counting elements per window, then reducing to the max per job; OR - A
KeyedProcessFunctionwith per-key state{currentCount, maxCount, lastSubmitTs}and an event-time timer atlastSubmitTs + 10min; on timer fire, close the stage (maxCount = max(maxCount, currentCount); currentCount = 0).
- Flink
2.3 Job-finish determination (broadened terminal set + end-of-input flush)
A job should be treated as finished when ANY of the following is true:
- A job_events record for that
jobIdarrives with a terminal event type. Treat the following as terminal:FINISH (4),KILL (5),FAIL (3),EVICT (2),LOST (6). In practice 3/4/5 are the dominant terminal signals; including 2/6 avoids missing edge-case completions.SUBMIT (0),SCHEDULE (1),UPDATE_PENDING (7),UPDATE_RUNNING (8)are NOT terminal. - The input stream has been fully consumed and the watermark advances to
Long.MAX_VALUE(MAX_WATERMARK). On bounded input this acts as an implicit terminal signal: any job that had SUBMIT activity but never received an explicit terminal job_event should still be flushed. This prevents dropping jobs whose completion was never recorded in the sliced input.
Do NOT infer completion solely from a 10-minute task-SUBMIT gap — the 10-minute rule only closes a stage, not the job. Use the session timer to roll currentStageCount into maxStageCount, and use the terminal job_event (or end-of-input) to decide when to emit.
Union the task-event and job-event streams keyed by jobId, or connect them with connect(...).keyBy(jobId, jobId) so a CoProcessFunction/KeyedCoProcessFunction sees both. Maintain per-job state: maxStageCount, currentStageCount, jobFinished (boolean), lastSubmitTs, emitted (boolean).
2.4 Output eligibility
- Emit
(jobId, maxStageCount)once per job, when both of these have happened:- The job has been marked finished per 2.3 (either a terminal job_event in {2,3,4,5,6} arrived, OR end-of-input/
MAX_WATERMARKwas reached), AND - Any still-open stage has been folded into
maxStageCount. If a terminal job_event arrives while a stage is still open (i.e. the 10-minute timer has not fired), flush it immediately:maxStageCount = max(maxStageCount, currentStageCount); currentStageCount = 0.
- The job has been marked finished per 2.3 (either a terminal job_event in {2,3,4,5,6} arrived, OR end-of-input/
- Jobs with zero SUBMIT events must not be emitted (
maxStageCount == 0means no stage ever existed). - Emit exactly one line per finished job. Guard with an
emittedflag (or clear per-key state after emission) to prevent duplicates when both a terminal job_event and end-of-input occur. - Do not require any specific arrival order between the terminal job_event and the stage-closing timer; whichever happens later triggers emission.
2.5 End-of-stream flushing
- Register an on-timer handler keyed to
Long.MAX_VALUE(or implementprocessWatermark/closelogic) so that when the bounded input finishes:- Any open stage for every job is closed (fold
currentStageCountintomaxStageCount). - Every job that has at least one SUBMIT is emitted exactly once, regardless of whether a terminal job_event was observed.
- Any open stage for every job is closed (fold
- If running in bounded/batch mode, calling
env.execute()afterDataStream#sinkTois sufficient; make sure the sink is aFileSinkorwriteAsText(path).setParallelism(1)so output is a single local file with one tuple per line.
Workflow 3 — Output formatting
- Each line must be exactly
(jobId,maxStageCount)with no spaces, e.g.(6253031646,42). - Use
Tuple2<Long, Long>and override itstoString()via aMapFunctionthat returns"(" + f0 + "," + f1 + ")"to avoid Flink's default"(a,b)"which already matches — but double-check no extra whitespace. Write with parallelism 1 to theoutputpath argument.
Workflow 4 — Argument parsing
- Accept
--task_input,--job_input,--outputviaParameterTool.fromArgs(args)and pass to your sources/sink. Overwrite the output file if it exists (FileSinkwithOutputFileConfigor delete beforehand).
Transfer patterns from retrieved examples
The retrieved Python examples (CSV parsing with pd.read_csv/csv.reader, filtering by prefix, writing CSV outputs) reinforce two reusable habits applicable here:
- Parse strictly by column index/name and coerce types early — do the same when splitting CSV lines in Java (
line.split(",", -1), thenLong.parseLongwith guards for empty fields likemachine_id). - Separate IO from logic — keep the gunzip+CSV parsing in a dedicated source function and keep stage/finish logic in
KeyedCoProcessFunction, mirroring how the examples isolate loading from counting.
Self-check before finishing
- [ ] Only
event_type==0task rows increment stage counts; repeats counted separately. - [ ] Stage closes strictly on
>= 10 minevent-time inactivity. - [ ] Job marked finished on job_events terminal types {2,3,4,5,6} OR on end-of-input (
MAX_WATERMARK), not from task inactivity alone. - [ ] On terminal job_event arrival, any still-open stage is flushed into
maxStageCountbefore emission. - [ ] Output emitted ONLY for jobs with ≥1 SUBMIT; exactly one line each; format
(jobId,count)with no spaces. - [ ] End-of-input watermark flushes open stages AND emits any finished-but-unemitted jobs; emission is idempotent (guarded by an
emittedflag). - [ ] Writes to the
--outputpath as a single local text file.
04 同仓库更多 Skills
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms