AGENT SKILL

training

Configure, validate, launch, and resume Motus three-stage training on a single node or SLURM cluster without confusing fine-tuning, checkpoint resume, or backbone initialization.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
training
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Configure, validate, launch, and resume Motus three-stage training on a single node or SLURM cluster without confusing fine-tuning, checkpoint resume, or backbone initialization.
disable-model-invocation
true

02 资源文件 4

03 SKILL.md 正文

Motus training

Use this skill when a user needs to prepare a Motus training configuration, check it safely, construct a torchrun/DeepSpeed command, or reason about resuming a run. It covers the three-stage intent, not dataset conversion or inference.

  • Read configuration.md for the config contract, derived values, and checkpoint-mode rules.
  • Read workflows.md for single-node, SLURM, and checkpoint export procedures. Launchers in the repository are reference-only: they are long-running and may submit external cluster jobs.
  • Read troubleshooting.md when validation, imports, CUDA, flash-attn, NCCL, logging, data, or checkpoint loading fails.
  • For dataset directory/layout preparation, use the sibling data-preparation skill. For inference, use model-inference.

Operating procedure

  1. Identify intent before editing YAML. Select pretrain, finetune, or resume. Record whether this is Stage 1 VGM training, Stage 2 Motus pretraining with latent actions, or Stage 3 target-robot SFT with actions. A checkpoint resume is a fourth execution mode, not a new training stage.
  2. Choose an example config. Use configs/latent_action.yaml for the explicit training_mode: pretrain latent-action pipeline; use the embodiment config matching the dataset (robotwin, ac_one, aloha_agilex_2, or lerobot) for fine-tuning. Replace all example paths with paths valid on the target machine.
  3. Run safe checks before any launch. Parse YAML, check the required sections and enum-like fields, calculate action_chunk_size, verify local dataset/checkpoint/VAE/config paths where available, and check GPU count, CUDA, bfloat16, DeepSpeed, and flash-attn. A parser/import check is allowed on CPU; model construction, VAE access, and training require CUDA and the large WAN/VLM checkpoints.
  4. Make checkpoint intent mutually clear. Never set a resume checkpoint and a fine-tune checkpoint accidentally. Resume loads the complete Accelerator state and continues its step number; fine-tune partially initializes a finetune model and skips action input/decoder weights; scratch loads WAN and VLM backbones from model.*.checkpoint_path.
  5. Review the rendered command. Check node count, GPUs per node, master address/port, config path, DeepSpeed JSON, logging mode, output/checkpoint directory, and the absence of accidental shell comments after \\. Prefer a short, bounded smoke configuration before an expensive run.
  6. Launch only with explicit approval. torchrun, sbatch, srun, WAN/VLM downloads, and full training are external, long-running, or expensive operations. This skill explains and constructs them but does not silently execute them. Use the repository launchers only as templates; do not copy their placeholder paths into a public skill or run them as validation.
  7. Verify expected signals. Rank 0 should report the selected dataset and training mode, checkpoint directory, logging backends, model creation, dataloader creation, and step/loss lines. A saved checkpoint directory should contain Accelerator/DeepSpeed state and the filtered config.json. Use scripts/export_config_json.py to create that JSON deliberately for an existing checkpoint directory.

Hard boundary

This skill does not prescribe dataset layouts, conversion, or inference CLI flags. It does not promise CPU training: Motus uses CUDA autocast and a very large WAN/VLM stack, and the documented training requirement is more than 80 GB VRAM (typically eight 80-GB GPUs for the supplied examples). Keep missing model checkpoints, datasets, optional packages, cluster topology, and unverified hardware as explicit blockers rather than guessing.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351