sprite-from-video
Walk, run or jump sprite loops from one still via generated video: keyed frames, clean seam.
- License 许可
- MIT
- 最近更新
- 2026-11-10
- Stars
- ★ 2
- Forks
- 0
01 前言 Front Matter
- name
- sprite-from-video
- description
- Walk, run or jump sprite loops from one still via generated video: keyed frames, clean seam.
02 资源文件 6
- 参考文档reference/loop-detection.md12.5 KB
- 参考文档reference/pipeline.md15.1 KB
- 参考文档reference/video-prompts.md7.3 KB
- 脚本scripts/video_canvas.py20.6 KB
- 脚本scripts/video_frames.py46.1 KB
- 脚本scripts/video_loop.py75.8 KB
03 SKILL.md 正文
sprite-from-video — one still, one clip, one clean cut
An image model draws a pose; a video model moves it. This skill takes one approved still of a character on a chroma key, has a video model perform one motion in place on it, and cuts that clip into frames a sprite pipeline can ship: transparent, on one canvas, with the loop seam or the one-shot's ends found by measurement rather than by eye.
Start here — symptom → where to look
| Symptom | Go to |
|---|---|
"Turn this still into a walk loop"; a run, or an idle with a weight shift, from a video (a breathing idle: codex-visual breathe.py) |
reference/pipeline.md §1–6 |
| Which canvas: the jump leaves the top of the frame, the attack hits the side | reference/pipeline.md §3; video_canvas.py --state/--shape |
video_canvas.py exit 2 or 3: the still's corners disagree or are not a key |
reference/loop-detection.md §7 |
| The clip came back frozen, zooming, panning or walking out of frame | reference/video-prompts.md §1–2, then re-roll |
| Green rim on the frames, or key colour left at the edge | reference/loop-detection.md §7 (video_frames.py exit 3, residual) |
| A limb or the head leaves the frame | reference/loop-detection.md §7 (exit 3, subject) — a taller or wider canvas |
| A frame keyed to nearly nothing transparent | reference/loop-detection.md §7 (exit 2) — the field is not the key |
| The loop pops at the seam | reference/loop-detection.md §3, §8 (seam over 2.0) |
| The clip is half a stride; the legs swap at the wrap | reference/loop-detection.md §2 (half-step guard) |
The loop holds two strides, or verdict.ok is false |
reference/loop-detection.md §2, §8 (exit 3) |
| No period found in a walk | reference/loop-detection.md §8 — the motion does not repeat: re-roll |
| The character drifts across the loop | reference/loop-detection.md §5 (drift ramp) |
| "Cut the jump out of this clip"; the jump cut starts mid-crouch or ends mid-landing | reference/pipeline.md §6 — --mode once |
| The animation plays too fast or too slow in Unity | reference/pipeline.md §7 — fps_hint, never 24 |
| The walk slides in Unity's frame audit | reference/pipeline.md §7 (register ground) |
The clip is raw material, never the asset. Every stage has a gate, and a clip that fails a gate is re-rolled (a new seed, a better prompt) rather than repaired. Nothing here interpolates, retimes or paints frames.
Rules the route is built on
- A loop comes from plain image-to-video plus period detection. Generate the motion running
continuously in place (a walk on a treadmill), then
video_loop.pyfinds one cycle in it and the start whose wrap is smoothest. - First frame = last frame does not make a loop. First/last-frame video with the same still at both ends comes back almost frozen: the model satisfies both ends by barely moving. Do not use it for any loop.
- An idle breath is
breathe.py(skillcodex-visual) on the keyed still — pixel-exact and closed by construction. An image-to-video idle cut byvideo_loop.py --state idleis the second choice, for an idle that has to shift its weight. - Jump, launch and attack are one-shots. They are generated once from standing and cut with
--mode once(standing → action → settled) or--mode oneshot(matching end poses). - The painted key is not the declared key. A video model paints "#00FF00" a dozen or more
units off, and differently per clip;
video_frames.pyreads the painted colour from the first frame's border and erases around it as well as around the exact key. - H3 always writes an audio track.
video_frames.pyextracts pictures only; never keep or ship the sound.
Files
| Path | Role |
|---|---|
scripts/video_canvas.py |
still → the canvas the motion needs (square, tall for a jump, wide for an attack), field repainted to the exact key, sized for H3; prints the --set 7.width=… --set 7.height=… line |
scripts/video_frames.py |
clip → every frame keyed to RGBA, with a key-flatness report and an edge-contact gate |
scripts/video_loop.py |
keyed frames → one loop, one-shot or once cut, drift removed, subsampled, on one canvas, plus frames.json and a preview GIF |
reference/pipeline.md |
the five stages with exact commands, which route each motion takes, the handoff |
reference/video-prompts.md |
how to word an in-place clip for H3, and the worked prompts |
reference/loop-detection.md |
the distance matrix, period, seam, one-shot and once rules, every exit code and its fix |
PY=$HOME/.codex/imagegen-venv/bin/python # numpy + Pillow
S=<repo>/skills/sprite-from-video/scripts
"$PY" "$S/video_canvas.py" --selftest && "$PY" "$S/video_frames.py" --selftest && "$PY" "$S/video_loop.py" --selftest
video_frames.py also needs ffmpeg and ffprobe on PATH (5.1 or later for
-fps_mode passthrough). Missing → stop and ask the user; never install them.
video_canvas.py imports video_frames.py from its own folder for the keyer, so the two
travel together.
The route
| Stage | Run | Gate before the next stage |
|---|---|---|
| 1. Canvas | video_canvas.py STILL --state walk --out CANVAS.png |
exit 0: corners flat (±24 per channel), a chroma key, field repainted to the exact key |
| 2. Generate | comfyui-asset-generation's comfy_asset_client.py with templates/minimax_h3_i2v_api.json, the canvas uploaded as 1.image, the prompt as 7.prompt, 7.width/7.height from stage 1, 7.length=124, a recorded 8.noise_seed |
terminal /history, the MP4 downloaded and probed; one GPU job at a time |
| 3. Key | video_frames.py CLIP --out-dir WORK |
exit 0: every frame transparent enough to be the key, nothing but the floor touching the frame edge |
| 4. Cut | video_loop.py WORK --state walk --name runner_walk --out-dir OUT |
exit 0: verdict.ok, seam ≤ 2.0 mean steps, the GIF watched once |
| 5. Hand off | unity-2d-sprites builds the clip from the frame folder at fps_hint; art-asset-registry ingests frames.json |
the Unity acceptance reads; the ledger record |
The commands in full, the per-motion choices and a worked runner walk: reference/pipeline.md.
Which cut for which motion (--state sets the search window; --mode the kind of cut):
| Motion | Canvas --state |
Prompt | Cut |
|---|---|---|---|
| walk, run | walk / run (square) |
treadmill, continuous | --state walk / run, auto → loop |
| idle breath | — | — | codex-visual breathe.py on the keyed still |
| idle with a weight shift | idle (square) |
in place, continuous | --state idle (no periodicity gate) |
| jump, launch | jump (tall 3:4, head-room 0.34) |
one jump from standing, still for the last second | --state jump --mode once |
| attack, swipe | attack (wide 16:9, lead 0.28) |
one attack from standing, back to stance | --state attack --mode once, or oneshot for a self-contained beat |
Outputs
video_loop.py writes OUT/NAME/NAME_000.png … on one canvas (the union of every frame's box
plus --margin, bottom-aligned, never resampled) and OUT/NAME/frames.json beside them, with
OUT/NAME.gif (the preview) and OUT/NAME.report.json (the whole decision) outside the frame
folder, so a Unity folder import picks up frames only. frames.json is frames/1 (the repo's
schemas/frames-1.json; codex-visual → reference/animation-scripts.md §4): canvas,
pivot_norm (foot centre, from the bottom-left), key_color (the declared key; a bare PNG folder
needs --key-color), fps_hint (output frames / cycle seconds), loop, hold_last, register
(mode ground with every frame seated on one feet line when it spreads by at most max(3 px, 2 % of
the body), keep for vertical travel; reference_core_height_px, foot_margin_px),
verdict{ok, faults}, producer{skill: sprite-from-video, script: video_loop.py}, the clip's
sha256, and per frame its source_frame.
Scope — what this skill does NOT do
- Running the generation — the server URL, uploads, the queue, model loading and
/free, the H3 templates and their node ids:comfyui-asset-generation. This skill decides the canvas, the prompt and the cut; that skill runs the job. - The still itself, sprite sheets, the art bible and the key choice — generating and keying
the base still,
pick_key.pyfor a key the subject does not contain, a grid sheet sliced into frames, a breathing idle from one frame (breathe.py):codex-visual. - Unity import — slicing, the shared pivot,
AnimationClip, loop vs one-shot playback, the frame audit:unity-2d-sprites. - The ledger — recording the clip, the seed and the frames as one asset with links:
art-asset-registry. - Choosing 2D frames over a rig for a character:
game-character-pipeline; 2D bones and skinning:unity-2d-animation. - Frame interpolation, repainting a frame, multi-direction sets: not done. A bad clip is re-rolled.
Related: comfyui-asset-generation → reference/minimax-h3.md (modes, frame grid, audio),
codex-visual → reference/animation-scripts.md (the frames contract, breathe.py),
unity-2d-sprites → reference/sprite-animation.md (the clip from a frame folder).
04 同仓库更多 Skills
unity-web-release
Ship a WebGL/WebGPU build: shrink the download, Brotli server headers, Safari and iPhone memory.
unity-vivox
Vivox voice and text chat: channels, positional voice, mute and push-to-talk, mic permission.
unity-urp-migration
Migrate a Built-in project to URP, or fix a half-moved one: pink materials, PPv2, old shaders.
unity-ui-ugui
Canvas UI (uGUI): CanvasScaler, RectTransform anchors, Layout Groups, ScrollView, UI built from C#.
unity-ui-toolkit
UI Toolkit (UXML, USS, UIDocument): flex layout, binding, custom elements, editor windows.
unity-ugs
Unity Gaming Services API: which package, init and sign-in order, Cloud Save, Cloud Code, Remote Config.