training
Configure, validate, launch, and resume Motus three-stage training on a single node or SLURM cluster without confusing fine-tuning, checkpoint resume, or backbone initialization.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- training
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Configure, validate, launch, and resume Motus three-stage training on a single node or SLURM cluster without confusing fine-tuning, checkpoint resume, or backbone initialization.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/configuration.md11.5 KB
- 参考文档references/troubleshooting.md11.7 KB
- 参考文档references/workflows.md11.0 KB
- 脚本scripts/export_config_json.py3.7 KB
03 SKILL.md 正文
Motus training
Use this skill when a user needs to prepare a Motus training configuration, check
it safely, construct a torchrun/DeepSpeed command, or reason about resuming a
run. It covers the three-stage intent, not dataset conversion or inference.
- Read configuration.md for the config contract, derived values, and checkpoint-mode rules.
- Read workflows.md for single-node, SLURM, and checkpoint export procedures. Launchers in the repository are reference-only: they are long-running and may submit external cluster jobs.
- Read troubleshooting.md when validation, imports, CUDA, flash-attn, NCCL, logging, data, or checkpoint loading fails.
- For dataset directory/layout preparation, use the sibling data-preparation skill. For inference, use model-inference.
Operating procedure
- Identify intent before editing YAML. Select
pretrain,finetune, or resume. Record whether this is Stage 1 VGM training, Stage 2 Motus pretraining with latent actions, or Stage 3 target-robot SFT with actions. A checkpoint resume is a fourth execution mode, not a new training stage. - Choose an example config. Use
configs/latent_action.yamlfor the explicittraining_mode: pretrainlatent-action pipeline; use the embodiment config matching the dataset (robotwin,ac_one,aloha_agilex_2, orlerobot) for fine-tuning. Replace all example paths with paths valid on the target machine. - Run safe checks before any launch. Parse YAML, check the required
sections and enum-like fields, calculate
action_chunk_size, verify local dataset/checkpoint/VAE/config paths where available, and check GPU count, CUDA, bfloat16, DeepSpeed, and flash-attn. A parser/import check is allowed on CPU; model construction, VAE access, and training require CUDA and the large WAN/VLM checkpoints. - Make checkpoint intent mutually clear. Never set a resume checkpoint and
a fine-tune checkpoint accidentally. Resume loads the complete Accelerator
state and continues its step number; fine-tune partially initializes a
finetunemodel and skips action input/decoder weights; scratch loads WAN and VLM backbones frommodel.*.checkpoint_path. - Review the rendered command. Check node count, GPUs per node, master
address/port, config path, DeepSpeed JSON, logging mode, output/checkpoint
directory, and the absence of accidental shell comments after
\\. Prefer a short, bounded smoke configuration before an expensive run. - Launch only with explicit approval.
torchrun,sbatch,srun, WAN/VLM downloads, and full training are external, long-running, or expensive operations. This skill explains and constructs them but does not silently execute them. Use the repository launchers only as templates; do not copy their placeholder paths into a public skill or run them as validation. - Verify expected signals. Rank 0 should report the selected dataset and
training mode, checkpoint directory, logging backends, model creation,
dataloader creation, and step/loss lines. A saved checkpoint directory
should contain Accelerator/DeepSpeed state and the filtered
config.json. Use scripts/export_config_json.py to create that JSON deliberately for an existing checkpoint directory.
Hard boundary
This skill does not prescribe dataset layouts, conversion, or inference CLI flags. It does not promise CPU training: Motus uses CUDA autocast and a very large WAN/VLM stack, and the documented training requirement is more than 80 GB VRAM (typically eight 80-GB GPUs for the supplied examples). Keep missing model checkpoints, datasets, optional packages, cluster topology, and unverified hardware as explicit blockers rather than guessing.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).