skilltta-task-guidance
Task-specific operational guidance synthesized at test time from retrieved training examples.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- skilltta-task-guidance
- description
- Task-specific operational guidance synthesized at test time from retrieved training examples.
03 SKILL.md 正文
SKILL.md
When to use
Use this skill when the task requires analyzing a video to produce two paired artifacts describing per-frame camera egomotion and dynamic-object segmentation, sampled at a fixed fps. The binding contract requires:
- Reading a video from a given path and sampling frames at the specified fps.
- Producing a JSON that maps half-open frame intervals (
"start->end") to a list of motion labels from a fixed allowed vocabulary (stay / dolly / pan / tilt / roll variants). - Producing an
.npzarchive containing ashapearray[H, W]plus, for every sampled framei, three CSR sparse arrays keyed asf_{i}_data,f_{i}_indices,f_{i}_indptrthat encode a binary dynamic-object mask.
Before acting, gather: video dimensions and total frame count, actual sampling indices given the fps, the exact label vocabulary the evaluator expects, the CSR sparse layout convention (per-row indptr of length H+1), and whether any pretrained motion/segmentation models are available in the environment. Retrieved BigCodeBench examples only illustrate generic JSON I/O patterns; they do not constrain schema here.
Possible Failure Modes
- Using the wrong sampling rate or mis-mapping sampled indices back to original frame indices; the interval keys must index sampled frames (0, 1, 2, ...), not raw video frames.
- Emitting interval keys as closed ranges or off-by-one (e.g.,
"0->0"for a single frame) instead of the required half-open"i->i+1". - Writing motion labels with wrong casing, punctuation, or synonyms (e.g., "pan_left", "PanLeft", "Zoom In") instead of the exact allowed strings.
- Storing labels as a string instead of a list; the value for each interval must be a list even when there is only one label.
- Omitting the
shapekey in the.npz, or storing it as a tuple/scalar instead of a length-2 array[H, W]. - Building CSR incorrectly: e.g., using COO or flat boolean arrays, forgetting that
indptrmust have lengthH+1, or swapping rows/columns soindicesindexes rows instead of columns. - Missing frames in the
.npz(must have entries for every sampled frame index used in the JSON) or an index mismatch between the JSON intervals and the mask frames. - Producing masks at a different resolution than the value written under
shape. - Silently skipping frames when the video has fewer frames than expected, causing key gaps.
- Overwriting or corrupting unrelated files at
/root/; write only the two required paths.
Possible procedures
- Load the video (e.g., with OpenCV/decord/ffmpeg), read fps and frame count, and compute sampled frame timestamps at
fps=5. Deduplicate and index them as0..N-1. - For egomotion: obtain per-sampled-frame motion using either a pretrained egomotion/optical-flow model or a classical pipeline (feature matching + essential matrix / homography decomposition). Convert the dominant motion component per frame into exactly one (or more) of the allowed labels. Default to
Staywhen motion magnitude is below a threshold. - Emit intervals: the simplest safe form is one entry per sampled frame as
"i->i+1": [label]. Optionally merge consecutive identical labels into longer intervals, but keep the half-open semantics. - For dynamic-object masks: run a segmentation/motion-segmentation approach (e.g., moving-object detector, ego-motion-compensated flow residual, or an off-the-shelf video object segmentation model). Produce a boolean
H×Wmask per sampled frame at a consistent resolution. - Convert each dense boolean mask to
scipy.sparse.csr_matrixand store.data,.indices,.indptrunder keysf_{i}_data,f_{i}_indices,f_{i}_indptr. Addshape=np.array([H, W]). Save withnp.savez(orsavez_compressed). - Cross-check: the set of frame indices
iused in the JSON intervals must match the set off_{i}_*groups in the.npz.
Decision points:
- If no dynamic objects are detected in a frame, still write an all-False CSR mask (valid empty
data/indices,indptrof zeros lengthH+1) rather than omitting the frame. - If motion estimation fails on a frame (e.g., too few features), fall back to
Stayrather than skipping. - If model dependencies are unavailable, prefer a lightweight classical pipeline over failing.
Verification Checklist
- [ ]
/root/pred_instructions.jsonexists, is valid JSON, and every key matches the pattern"<int>-><int>"withend == start + kfork >= 1and half-open semantics. - [ ] Every label is exactly one of:
Stay, Dolly In, Dolly Out, Pan Left, Pan Right, Tilt Up, Tilt Down, Roll Left, Roll Right, and each value is a JSON list. - [ ] Intervals cover the full range of sampled frame indices with no gaps or overlaps.
- [ ]
/root/pred_dyn_masks.npzloads withnp.load(..., allow_pickle=False)and contains ashapearray of length 2 with positive integers. - [ ] For every sampled frame index
ireferenced by the JSON, all three keysf_{i}_data,f_{i}_indices,f_{i}_indptrare present,indptrhas lengthH+1,indptr[0] == 0,indptr[-1] == len(data) == len(indices), andindicesvalues are in[0, W). - [ ] Reconstructing a mask via
csr_matrix((data, indices, indptr), shape=(H,W)).toarray().astype(bool)yields a boolean array with the declared shape. - [ ] Sampling rate used is
fps = 5; the number of sampled frames matchesfloor(duration * 5)(or the chosen consistent convention) and is used consistently across both files. - [ ] Only the two required output files under
/root/are created/modified; no unrelated state is touched. - [ ] Retrieved BigCodeBench examples were used only as generic I/O inspiration; none of their filenames, keys, or schemas leaked into the outputs.
- [ ] On any per-frame failure, a safe fallback (Stay label / empty mask) was emitted so no key is missing.
04 同仓库更多 Skills
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms