training

Training

Use this sub-skill when the task is to plan or debug training with train.py without starting an expensive run. It covers the training CLI, data input modes, safe command construction, loss/scheduler smoke checks, logs, checkpoints, device flags, TensorBoard, and recoverable training setup failures.

Do not use this sub-skill to create training data files, translate from a checkpoint, or explain Transformer internals beyond training-facing constructor flags. Route those tasks to the data-preparation, translation, or model-architecture sub-skills.

Fast routes

• Build a command without running training: use scripts/build_training_command.py, then confirm the details in references/cli-reference.md.
• Validate training helpers and a tiny model/scheduler path before a long run: use scripts/training_smoke_check.py.
• Choose between all-in-one pickle training and BPE-prefix training: read references/workflows.md.
• Recover from missing -output_dir, stale README flags, CUDA failures, weight-sharing assertions, or TensorBoard import problems: read references/troubleshooting.md.

Safe operating pattern

1. Decide the data mode:
   • all-in-one pickle: -data_pkl DATA.pkl only;
   • BPE files: -data_pkl BPE_VOCAB.pkl -train_path TRAIN_PREFIX -val_path VAL_PREFIX where .src and .trg files exist for both prefixes.
2. Always include -output_dir; train.py raises if it is omitted.
3. Prefer -no_cuda for CPU-only planning and smoke checks. Omit it only when CUDA is available and desired.
4. Use scripts/build_training_command.py to print the exact command. It is safe by default and does not execute training.
5. Run scripts/training_smoke_check.py --repo-root <checkout> in an environment with the legacy training dependencies when you need to verify import, loss, model forward, and ScheduledOptim behavior.
6. Treat full training as a long-running user action requiring prepared data, a writable output directory, and explicit approval.

Key runtime facts

• train.py uses torchtext.data.Field, Dataset, BucketIterator, and TranslationDataset, so it expects the legacy torchtext API rather than modern torchtext datapipes.
• The training loop writes train.log and valid.log inside -output_dir, overwriting existing files with the same names at run start.
• With default -save_mode best, the checkpoint is OUTPUT_DIR/model.chkpt. With -save_mode all, checkpoint files are written in the process working directory, not in -output_dir.
• ScheduledOptim wraps Adam with the Transformer warmup schedule; it updates the learning rate on every step_and_update_lr() call.
• The README examples include a historical -log flag, but the current train.py parser does not accept it. Omit -log from generated commands.