single-modality
Operate InternVideo2 single-modality visual workflows for pretraining, finetuning, probing, distillation, model selection, datasets, launch adaptation, and troubleshooting.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- single-modality
- license
- Apache 2.0
- metadata
- { "disco-role": "operating", "source-branch": "InternVideo2-single_modality", "source-commit": "3965eef16e2dadd0ea6c8d0cc29c8a3039df52e3" }
- description
- Operate InternVideo2 single-modality visual workflows for pretraining, finetuning, probing, distillation, model selection, datasets, launch adaptation, and troubleshooting.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/configuration.md10.3 KB
- 参考文档references/troubleshooting.md10.0 KB
- 参考文档references/workflows.md9.4 KB
- 脚本scripts/build_single_modality_command.py3.9 KB
03 SKILL.md 正文
InternVideo2 Single-Modality Operating Skill
Use this sub-skill when a user asks about the visual-only single_modality branch of InternVideo2: masked video pretraining, action-classification finetuning, linear or attentive probing, distillation from larger teachers, model-zoo choice, dataset/weight paths, SLURM or DeepSpeed launch adaptation, or single-modality failure diagnosis.
Do not use this sub-skill for video-text/audio retrieval, chat/video-MLLM work, InternVideo3, InternVideo-Next, or InternVid dataset acquisition beyond the action-recognition annotation layouts needed here. Route those to the appropriate sibling sub-skill or root repo guidance.
Operating protocol
- Classify the requested workflow. Choose one of: pretraining, distillation, full finetuning, linear probing, attentive probing, or eval-only classification. Use references/workflows.md for entry points and command shapes.
- Collect required user inputs before executing anything expensive. Confirm dataset split root or pretraining CSV, checkpoint/model root, desired model size, launcher (
srun,torchrun, or plainpython), GPU count, precision, and whether the user is asking for command construction only or an actual run. - Check prerequisites. Use references/configuration.md to verify Python/CUDA dependencies,
INTERNVIDEO2_DATA_PATH,INTERNVIDEO2_MODEL_PATH, annotation CSV layout, teacher weight locations, and model registry names. Full training/evaluation requires user-provided datasets, checkpoints, CUDA-capable dependencies, and cluster resources. - Generate commands safely. Prefer the bundled helper scripts/build_single_modality_command.py to render a dry-run command. The helper prints command text only; it has no training side effects. Review paths, GPUs, batch size, DeepSpeed flags, and dataset class before running.
- Adapt launchers deliberately. Source workflows are SLURM-first and usually use DeepSpeed with BF16. For local
torchrun, preserve distributed environment variables and port selection. For single-process debugging, remove--enable_deepspeed,--dist_eval, and large batch sizes unless the user explicitly prepared that stack. - Diagnose by symptom. Use references/troubleshooting.md for missing CUDA extensions, teacher checkpoint failures, dataset parsing errors, stale source shell scripts, class-head mismatches, distributed hangs, and OOM recovery.
Quick route table
| User intent | Use | Key inputs |
|---|---|---|
| Train Stage1 1B/6B visual model on K-Mash-style data | run_pretraining.py recipe in workflows |
pretraining CSV, InternVL visual teacher, VideoMAEv2-g teacher, large multi-GPU launch |
| Distill S/B/L from InternVideo2 Stage2-1B | run_distill.py recipe in workflows |
K-Mash CSV, teacher checkpoint registry patched to portable paths, model size |
| Full tune on Kinetics/K400/K600/K700/MiT/SSV2/ANet/HACS | run_finetuning.py recipe in workflows |
split root with train.csv, val.csv, test.csv; checkpoint; classes; frames/crops |
| Linear probe or attentive probe | run_linear_probing.py recipe in workflows |
checkpoint, dataset class, open_clip_projector/open_block_num, frame schedule |
| Select checkpoint/model family | configuration model table | target dataset, model scale, stage, available weights |
| Fix a run failure | troubleshooting | first error line, command, environment summary, dataset/checkpoint existence |
Non-negotiable boundaries
- Do not submit source shell scripts blindly. They are cluster-specific and some contain stale names or non-portable checkpoint assumptions; distill their options into a reviewed command first.
- Do not claim CPU validation proves training readiness. The selected verification for this skill is static/helper-script verification; native training and evaluation remain optional heavyweight GPU/SLURM work.
- Do not expose private machine paths, local environment prefixes, or credentials in generated commands or reports.
- Do not modify user code to patch checkpoint paths or loader bugs without explicit user approval. If a patch is needed, explain the exact portable change and why.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).