dyn-object-masks
Generate dynamic-object binary masks after global motion compensation, route by scene type, output CSR sparse format, with proof-of-execution overlays.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- dyn-object-masks
- description
- Generate dynamic-object binary masks after global motion compensation, route by scene type, output CSR sparse format, with proof-of-execution overlays.
03 SKILL.md 正文
When to use
- Detect moving objects in scenes with camera motion; produce sparse masks aligned to sampled frames.
Scene-type routing (do this first)
A 2D affine (or homography) cannot absorb parallax: on walking POV / dashcam scenes with foreground objects and distant background, the single global model fits one depth plane and leaves large residuals on the other, so static near-field objects light up as "dynamic". Decide the pipeline before differencing:
# Signals already available from egomotion-estimation per pair:
# s["dx"], s["dy"], s["scale"], s["resid_med"], s["resid_mad"]
translation_dominated = any(abs(s["dx"]) > abs(s["scale"]-1) * W or
abs(s["dy"]) > abs(s["scale"]-1) * H
for s in sigs)
high_resid = np.median([s["resid_mad"] for s in sigs]) > RESID_PARALLAX_FLOOR
parallax_heavy = translation_dominated and high_resid
Decision rule:
- If
parallax_heavy: use the flow-residual pipeline — seereferences/parallax-robust-masks.md. Do not rely on affine warped-diff. - Otherwise (rotation-dominated, tripod-ish, or small motions): use the affine warped-diff pipeline below.
Read references/parallax-robust-masks.md when the camera is
translation-dominated and per-pair residual spread is high. Skip when
the camera is near-stationary or purely rotational.
Affine warped-diff workflow (non-parallax path)
- Global alignment: warp previous gray frame to current using estimated affine/homography.
- Valid region: also warp an all-ones mask to get
validpixels, avoiding border fill. - Difference + adaptive threshold:
diff = abs(curr - warp_prev); ondiff[valid]compute median + 3×MAD; use a reasonable minimum threshold to avoid triggering on noise. - Morphology + area filter: open then close; keep connected components with area in
[min_area, max_area]. See False-negative guardrails for upper-bound sizing. - Temporal persistence: a true moving object usually appears across ≥2 consecutive sampled frames; see False-negative guardrails for stride-aware persistence.
- CSR encoding: for final bool mask
rows, cols = nonzero(mask)indices = cols.astype(int32);data = ones(nnz, uint8)counts = bincount(rows, minlength=H);indptr = cumsum(counts, prepend=0)- store as
f_{i}_data/indices/indptr
Code sketch
warped_prev = cv2.warpAffine(prev_gray, M, (W,H), flags=cv2.INTER_LINEAR, borderValue=0)
valid = cv2.warpAffine(np.ones((H,W),uint8), M, (W,H), flags=cv2.INTER_NEAREST)>0
diff = cv2.absdiff(curr_gray, warped_prev)
vals = diff[valid]
thr = max(MIN_DIFF, np.median(vals) + 3*1.4826*np.median(np.abs(vals - np.median(vals))))
raw = (diff>thr) & valid
m = cv2.morphologyEx(raw.astype(uint8)*255, cv2.MORPH_OPEN, k3)
m = cv2.morphologyEx(m, cv2.MORPH_CLOSE, k7)
n, cc, stats, _ = cv2.connectedComponentsWithStats(m>0, connectivity=8)
mask = np.zeros_like(raw, dtype=bool)
for cid in range(1, n):
area = stats[cid, cv2.CC_STAT_AREA]
w, h = stats[cid, cv2.CC_STAT_WIDTH], stats[cid, cv2.CC_STAT_HEIGHT]
aspect = max(w, h) / max(1, min(w, h))
if min_area <= area <= max_area and aspect <= MAX_ASPECT:
mask |= (cc == cid)
False-negative guardrails
Stacked suppressors (MAD@large-K + tight max-area + aspect + strict persistence) can empty most frames while every single filter looks defensible in isolation. Size each filter against a plausible-object budget and the sampling stride:
- Max-area:
max_area ≥ 0.25 * H * W(a close-up car or pedestrian easily exceeds 8% of frame). Let aspect-ratio / compactness do the façade-rejection work; do not use max-area for that. - Persistence at large stride: a raw-mask overlap requirement is
incompatible with real object displacement when stride is high — the
object moves off its own previous footprint. Either (a) compare against
a flow-warped neighbor mask, (b) dilate the neighbor mask by the
expected displacement before intersecting, or (c) drop persistence
entirely when
stride >= STRIDE_PERSIST_DISABLE(e.g. ≥8). - MAD multiplier: keep
K = 3; raising to 4–5 is appropriate only when area/aspect/persistence have already been loosened.
# Default sizing — generic, not task-specific
MAX_AREA_FRAC = 0.25 # of H*W; raise if objects can be larger
MAX_ASPECT = 6.0
K_MAD = 3.0
STRIDE_PERSIST_DISABLE = 8 # drop raw-mask persistence at/above this
Post-filter emptiness check (abort before shipping):
empty_frac = sum(not m.any() for m in masks) / len(masks)
max_cov = max(m.mean() for m in masks)
if empty_frac > 0.20 or max_cov < 0.01:
# Over-pruning. Loosen in reverse order: persistence → max_area → K_MAD,
# one at a time, and re-run before shipping. See parallax-robust-masks.md.
raise AssertionError(f"empty_frac={empty_frac:.2f}, max_cov={max_cov:.3f}: "
f"loosen filters and re-run")
Scene-grounded exit — required unconditionally (not only after loosening). The numeric band is tunable from inside this loop. Even a first-try pass is self-consistent, not scene-anchored, until an overlay is written to disk and a prose description names what each blob covers. The exit condition is band AND verdict AND file-on-disk AND prose.
Pick 2–3 non-empty mask frames (prefer the max-coverage frame + the first non-empty frame + one random non-empty frame). For each:
- Write an actual overlay PNG to disk; log the path in-trace.
- Open the overlay and write a prose verdict naming what the largest blob covers in English (e.g. "overlaps visible pedestrian with red jacket" / "covers static building façade — false positive").
- Assign the enum:
dynamic/false_positive/ambiguous.
# One-time scene-grounded gate; the exit condition is band AND verdict AND PNG AND prose.
verdicts = inspect_mask_overlays(masks, sampled_frames, sample_ids,
k_frames=3, out_dir=OVERLAY_DIR)
# verdicts: list[dict(idx, overlay_path, prose, verdict in {"dynamic","false_positive","ambiguous"})]
# Proof-of-execution checks — a verdict string without an on-disk PNG,
# or a bare enum without prose, counts as "not run".
for v in verdicts:
assert os.path.exists(v["overlay_path"]), (
f"overlay PNG missing at {v['overlay_path']} — gate not executed")
assert v.get("prose"), (
f"verdict for frame {v['sample_id']} has no prose — re-open and describe")
all_ok = all(v["verdict"] == "dynamic" for v in verdicts)
loosened_this_run = (K_MAD < 3.0) or (MAX_AREA_FRAC > 0.25) or persistence_disabled
cov_red_flag = loosened_this_run and max_cov > 0.30 # loosening-driven inflation
if (not all_ok) or cov_red_flag:
# Re-tighten the parameter most recently loosened and re-run — do NOT
# keep loosening to clear the numeric band. See parallax-robust-masks.md
# for the ordered re-tighten branch.
raise AssertionError(
f"scene-grounded gate failed: verdicts={verdicts}, "
f"max_cov={max_cov:.3f}, loosened={loosened_this_run}; retighten")
Read references/parallax-robust-masks.md for the overlay-inspection
helper, the per-frame verdict rubric with the English-vocabulary
checklist, and the re-tighten branch (ordered undo of the last loosening
step). Skip only when outputs are format-only graded and no retry
budget remains.
Self-check
- [ ] Scene-type routing decision recorded; parallax-heavy scenes use the flow-residual pipeline, not affine warped-diff.
- [ ] Masks only for sampled frames; keys match sampled indices.
- [ ]
shapestored as[H, W]int32;len(indptr)==H+1;indptr[-1]==indices.size. - [ ] Border fill not treated as foreground; threshold stats computed on valid region only.
- [ ] Area filter has both min and max bounds;
max_area ≥ 0.25*H*Wunless scene prior justifies tighter; aspect ratio bounded. - [ ] Persistence is stride-aware (warped/dilated neighbor, or disabled at large stride); not a raw-mask overlap requirement.
- [ ] Post-filter emptiness check passed:
empty_frac ≤ 0.20ANDmax_cov ≥ 0.01, or filters were loosened and the mask set re-generated. - [ ] Scene-grounded gate executed with proof-of-execution: 2–3 overlay PNG files exist on disk at paths logged in-trace; each verdict has a prose scene-content description (not a bare enum); all verdicts read "dynamic"; if any parameter was loosened and
max_cov > 0.30, the retighten branch was taken.
Common pitfalls
- Treating affine warped-diff as universal. It only works when the scene is near-planar or the camera motion is near-rotational. Translation-dominant motion on a depth-varying scene produces parallax residuals that look like dynamic objects — route to flow-residual instead.
- Ticked-but-not-executed scene-grounded gate. The self-check line for Gate 3 completes without the gate's substantive action occurring: no overlay PNG on disk, no prose description, just a verdict enum copied into a dict. A verdict without an on-disk PNG at a logged path and a prose line in English scene-content vocabulary counts as "not run" and blocks shipping.
- Treating the numeric band as the exit condition. The empty-frac/max-cov
band is tunable from inside the retry loop; a loosened pipeline can
satisfy the band by accepting static façade/parallax residual. The
exit condition is band AND scene-grounded verdict (with PNG + prose);
max_cov > 0.30after loosening is itself a red flag that triggers the retighten branch, not another loosening. - Raw-mask persistence at large sampling stride. The object moves off its own previous footprint between samples; use a flow-warped or displacement-dilated neighbor mask, or disable persistence entirely.
04 同仓库更多 Skills
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms
video-frame-extraction
Extract frames from video files and save them as images using OpenCV
openai-vision
Analyze images and multi-frame sequences using OpenAI GPT vision models
image-ocr
Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms