[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M758TXBF16WVM28QXR8XAWCT":3,"skill-same:01M758TXBF16WVM28QXR8XAWCT":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":51,"ownerHtmlUrl":52,"ownerBlogUrl":6,"ownerDescription":53,"updatedAt":54,"install":55,"resources":68},"01M758TXBF16WVM28QXR8XAWCT","distributed-llm-pretraining-torchtitan",null,"Pretrain LLMs at scale with PyTorch 4D parallelism.","lukemcqueen\u002Fsteadfaste-fleet","lukemcqueen","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F405240?v=4","01M757MHPAGRXTJAT46SKZP98R","Luke McQueen","01M757MHPZ573KP8DT9GB8SBQ7",4,"MIT","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fmlops\u002Ftorchtitan\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: distributed-llm-pretraining-torchtitan\ndescription: Pretrain LLMs at scale with PyTorch 4D parallelism.\nversion: 1.0.1\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Model Architecture, Distributed Training, TorchTitan, FSDP2, Tensor Parallel, Pipeline Parallel, Context Parallel, Float8, Llama, Pretraining]\n\n---\n\n# TorchTitan - PyTorch Native Distributed LLM Pretraining\n\n## Quick start\n\nTorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.\n\n**Installation**:\n```bash\n# From PyPI (stable)\npip install torchtitan\n\n# From source (latest features, requires PyTorch nightly)\ngit clone https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\ncd torchtitan\npip install -r requirements.txt\n```\n\n**Download tokenizer**:\n```bash\n# Get HF token from https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\npython scripts\u002Fdownload_hf_assets.py --repo_id meta-llama\u002FLlama-3.1-8B --assets tokenizer --hf_token=...\n```\n\n**Start training on 8 GPUs**:\n```bash\n# Configs are selected by name from the Python config registry\n# (torchtitan\u002Fmodels\u002Fllama3\u002Fconfig_registry.py), not by TOML path\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh\n```\n\n## Common workflows\n\n### Workflow 1: Pretrain Llama 3.1 8B on single node\n\nCopy this checklist:\n\n```\nSingle Node Pretraining:\n- [ ] Step 1: Download tokenizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Launch training\n- [ ] Step 4: Monitor and checkpoint\n```\n\n**Step 1: Download tokenizer**\n\n```bash\npython scripts\u002Fdownload_hf_assets.py \\\n  --repo_id meta-llama\u002FLlama-3.1-8B \\\n  --assets tokenizer \\\n  --hf_token=YOUR_HF_TOKEN\n```\n\n**Step 2: Configure training**\n\nIn torchtitan's current layout, run configs are defined in a Python **config registry**\n(`torchtitan\u002Fmodels\u002Fllama3\u002Fconfig_registry.py`) and selected by name via `CONFIG=\u003Cname>`\n(or `--config \u003Cname>`). To customize, register your own config in the registry, or override\nindividual fields on the command line (e.g. `--optimizer.lr 3e-4 --training.steps 1000`).\n\nThe equivalent settings for an 8B run look like this (shown as fields; set them in the\nregistry entry or as `--section.key value` overrides):\n\n```toml\n# fields for a llama3 8B run (register in config_registry.py or pass as --overrides)\n[job]\ndump_folder = \".\u002Foutputs\"\ndescription = \"Llama 3.1 8B training\"\n\n[model]\nname = \"llama3\"\nflavor = \"8B\"\nhf_assets_path = \".\u002Fassets\u002Fhf\u002FLlama-3.1-8B\"\n\n[optimizer]\nname = \"AdamW\"\nlr = 3e-4\n\n[lr_scheduler]\nwarmup_steps = 200\n\n[training]\nlocal_batch_size = 2\nseq_len = 8192\nmax_norm = 1.0\nsteps = 1000\ndataset = \"c4\"\n\n[parallelism]\ndata_parallel_shard_degree = -1  # Use all GPUs for FSDP\n\n[activation_checkpoint]\nmode = \"selective\"\nselective_ac_option = \"op\"\n\n[checkpoint]\nenable = true\nfolder = \"checkpoint\"\ninterval = 500\n```\n\n**Step 3: Launch training**\n\n```bash\n# 8 GPUs on single node (config selected by name from the registry)\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh\n\n# Override individual fields on the command line\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh --optimizer.lr 3e-4 --training.steps 1000\n\n# Or explicitly with torchrun (run_train.sh wraps this)\ntorchrun --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --module llama3 --config llama3_8b\n```\n\n**Step 4: Monitor and checkpoint**\n\nTensorBoard logs are saved to `.\u002Foutputs\u002Ftb\u002F`:\n```bash\ntensorboard --logdir .\u002Foutputs\u002Ftb\n```\n\n### Workflow 2: Multi-node training with SLURM\n\n```\nMulti-Node Training:\n- [ ] Step 1: Configure parallelism for scale\n- [ ] Step 2: Set up SLURM script\n- [ ] Step 3: Submit job\n- [ ] Step 4: Resume from checkpoint\n```\n\n**Step 1: Configure parallelism for scale**\n\nFor 70B model on 256 GPUs (32 nodes):\n```toml\n[parallelism]\ndata_parallel_shard_degree = 32  # FSDP across 32 ranks\ntensor_parallel_degree = 8        # TP within node\npipeline_parallel_degree = 1      # No PP for 70B\ncontext_parallel_degree = 1       # Increase for long sequences\n```\n\n**Step 2: Set up SLURM script**\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=llama70b\n#SBATCH --nodes=32\n#SBATCH --ntasks-per-node=8\n#SBATCH --gpus-per-node=8\n\nsrun torchrun \\\n  --nnodes=32 \\\n  --nproc_per_node=8 \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  -m torchtitan.train \\\n  --module llama3 --config llama3_70b\n```\n\n**Step 3: Submit job**\n\n```bash\nsbatch multinode_trainer.slurm\n```\n\n**Step 4: Resume from checkpoint**\n\nTraining auto-resumes if checkpoint exists in configured folder.\n\n### Workflow 3: Enable Float8 training for H100s\n\nFloat8 provides 30-50% speedup on H100 GPUs.\n\n```\nFloat8 Training:\n- [ ] Step 1: Install torchao\n- [ ] Step 2: Configure Float8\n- [ ] Step 3: Launch with compile\n```\n\n**Step 1: Install torchao**\n\n```bash\nUSE_CPP=0 pip install git+https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao.git\n```\n\n**Step 2: Configure Float8**\n\nIn the current torchtitan, Float8 is applied at config time via the `quantization`\nparameter in your `model_registry()` call inside the config registry (not via a\n`[quantize.linear.float8]` TOML section). Add a `Float8LinearConverter.Config`:\n\n```python\n# in torchtitan\u002Fmodels\u002Fllama3\u002Fconfig_registry.py (your model_registry(...) call)\nfrom torchtitan.components.quantization import Float8LinearConverter\n\nmodel_spec = model_registry(\n    \"8B\",\n    quantization=[\n        Float8LinearConverter.Config(\n            recipe_name=\"rowwise\",          # or \"rowwise_with_gw_hp\"\n            filter_fqns=[\"output\"],          # skip layers too small to benefit\n            model_compile_enabled=True,      # requires torch.compile for competitive perf\n        ),\n    ],\n)\n```\n\nEnable `torch.compile` in your run config too:\n```toml\n[compile]\nenable = true\ncomponents = [\"model\", \"loss\"]\n```\n\n**Step 3: Launch with compile**\n\n```bash\n# Float8 config is baked into the registered config; just select it and enable compile\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh --compile.enable\n```\n\n### Workflow 4: 4D parallelism for 405B models\n\n```\n4D Parallelism (FSDP + TP + PP + CP):\n- [ ] Step 1: Create seed checkpoint\n- [ ] Step 2: Configure 4D parallelism\n- [ ] Step 3: Launch on 512 GPUs\n```\n\n**Step 1: Create seed checkpoint**\n\nRequired for consistent initialization across PP stages:\n```bash\nNGPU=1 MODULE=llama3 CONFIG=llama3_405b .\u002Frun_train.sh \\\n  --checkpoint.enable \\\n  --checkpoint.create_seed_checkpoint \\\n  --parallelism.data_parallel_shard_degree 1 \\\n  --parallelism.tensor_parallel_degree 1 \\\n  --parallelism.pipeline_parallel_degree 1\n```\n\n**Step 2: Configure 4D parallelism**\n\n```toml\n[parallelism]\ndata_parallel_shard_degree = 8   # FSDP\ntensor_parallel_degree = 8       # TP within node\npipeline_parallel_degree = 8     # PP across nodes\ncontext_parallel_degree = 1      # CP for long sequences\n\n[training]\nlocal_batch_size = 32\nseq_len = 8192\n```\n\n**Step 3: Launch on 512 GPUs**\n\n```bash\n# 64 nodes x 8 GPUs = 512 GPUs\nsrun torchrun --nnodes=64 --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --module llama3 --config llama3_405b\n```\n\n## When to use vs alternatives\n\n**Use TorchTitan when:**\n- Pretraining LLMs from scratch (8B to 405B+)\n- Need PyTorch-native solution without third-party dependencies\n- Require composable 4D parallelism (FSDP2, TP, PP, CP)\n- Training on H100s with Float8 support\n- Want interoperable checkpoints with torchtune\u002FHuggingFace\n\n**Use alternatives instead:**\n- **Megatron-LM**: Maximum performance for NVIDIA-only deployments\n- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support\n- **Axolotl\u002FTRL**: Fine-tuning rather than pretraining\n- **LitGPT**: Educational, smaller-scale training\n\n## Common issues\n\n**Issue: Out of memory on large models**\n\nEnable activation checkpointing and reduce batch size:\n```toml\n[activation_checkpoint]\nmode = \"full\"  # Instead of \"selective\"\n\n[training]\nlocal_batch_size = 1\n```\n\nOr use gradient accumulation:\n```toml\n[training]\nlocal_batch_size = 1\nglobal_batch_size = 32  # Accumulates gradients\n```\n\n**Issue: TP causes high memory with async collectives**\n\nSet environment variable:\n```bash\nexport TORCH_NCCL_AVOID_RECORD_STREAMS=1\n```\n\n**Issue: Float8 training not faster**\n\nFloat8 only benefits large GEMMs. Filter small layers via the converter's `filter_fqns`:\n```python\nfrom torchtitan.components.quantization import Float8LinearConverter\n\nFloat8LinearConverter.Config(\n    # add \"auto_filter_small_kn\" to auto-skip layers too small to benefit\n    filter_fqns=[\"attention.wk\", \"attention.wv\", \"output\", \"auto_filter_small_kn\"],\n    model_compile_enabled=True,\n)\n```\n\n**Issue: Checkpoint loading fails after parallelism change**\n\nUse DCP's resharding capability:\n```bash\n# Convert sharded checkpoint to single file\npython -m torch.distributed.checkpoint.format_utils \\\n  dcp_to_torch checkpoint\u002Fstep-1000 checkpoint.pt\n```\n\n**Issue: Pipeline parallelism initialization**\n\nCreate seed checkpoint first (see Workflow 4, Step 1).\n\n## Supported models\n\n| Model | Sizes | Status |\n|-------|-------|--------|\n| Llama 3.1 | 8B, 70B, 405B | Production |\n| Llama 4 | Various | Experimental |\n| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |\n| GPT-OSS | 20B, 120B (MoE) | Experimental |\n| Qwen 3 | Various | Experimental |\n| Flux | Diffusion | Experimental |\n\n## Performance benchmarks (H100)\n\n| Model | GPUs | Parallelism | TPS\u002FGPU | Techniques |\n|-------|------|-------------|---------|------------|\n| Llama 8B | 8 | FSDP | 5,762 | Baseline |\n| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |\n| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |\n| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |\n\n## Advanced topics\n\n**FSDP2 configuration**: See [references\u002Ffsdp.md](references\u002Ffsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.\n\n**Float8 training**: See [references\u002Ffloat8.md](references\u002Ffloat8.md) for tensorwise vs rowwise scaling recipes.\n\n**Checkpointing**: See [references\u002Fcheckpoint.md](references\u002Fcheckpoint.md) for HuggingFace conversion and async checkpointing.\n\n**Adding custom models**: See [references\u002Fcustom-models.md](references\u002Fcustom-models.md) for TrainSpec protocol.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.06511\n- ICLR 2025: https:\u002F\u002Ficlr.cc\u002Fvirtual\u002F2025\u002Fposter\u002F29620\n- PyTorch Forum: https:\u002F\u002Fdiscuss.pytorch.org\u002Fc\u002Fdistributed\u002Ftorchtitan\u002F44\n\n","# TorchTitan - PyTorch Native Distributed LLM Pretraining\n\n## Quick start\n\nTorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.\n\n**Installation**:\n```bash\n# From PyPI (stable)\npip install torchtitan\n\n# From source (latest features, requires PyTorch nightly)\ngit clone https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\ncd torchtitan\npip install -r requirements.txt\n```\n\n**Download tokenizer**:\n```bash\n# Get HF token from https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\npython scripts\u002Fdownload_hf_assets.py --repo_id meta-llama\u002FLlama-3.1-8B --assets tokenizer --hf_token=...\n```\n\n**Start training on 8 GPUs**:\n```bash\n# Configs are selected by name from the Python config registry\n# (torchtitan\u002Fmodels\u002Fllama3\u002Fconfig_registry.py), not by TOML path\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh\n```\n\n## Common workflows\n\n### Workflow 1: Pretrain Llama 3.1 8B on single node\n\nCopy this checklist:\n\n```\nSingle Node Pretraining:\n- [ ] Step 1: Download tokenizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Launch training\n- [ ] Step 4: Monitor and checkpoint\n```\n\n**Step 1: Download tokenizer**\n\n```bash\npython scripts\u002Fdownload_hf_assets.py \\\n  --repo_id meta-llama\u002FLlama-3.1-8B \\\n  --assets tokenizer \\\n  --hf_token=YOUR_HF_TOKEN\n```\n\n**Step 2: Configure training**\n\nIn torchtitan's current layout, run configs are defined in a Python **config registry**\n(`torchtitan\u002Fmodels\u002Fllama3\u002Fconfig_registry.py`) and selected by name via `CONFIG=\u003Cname>`\n(or `--config \u003Cname>`). To customize, register your own config in the registry, or override\nindividual fields on the command line (e.g. `--optimizer.lr 3e-4 --training.steps 1000`).\n\nThe equivalent settings for an 8B run look like this (shown as fields; set them in the\nregistry entry or as `--section.key value` overrides):\n\n```toml\n# fields for a llama3 8B run (register in config_registry.py or pass as --overrides)\n[job]\ndump_folder = \".\u002Foutputs\"\ndescription = \"Llama 3.1 8B training\"\n\n[model]\nname = \"llama3\"\nflavor = \"8B\"\nhf_assets_path = \".\u002Fassets\u002Fhf\u002FLlama-3.1-8B\"\n\n[optimizer]\nname = \"AdamW\"\nlr = 3e-4\n\n[lr_scheduler]\nwarmup_steps = 200\n\n[training]\nlocal_batch_size = 2\nseq_len = 8192\nmax_norm = 1.0\nsteps = 1000\ndataset = \"c4\"\n\n[parallelism]\ndata_parallel_shard_degree = -1  # Use all GPUs for FSDP\n\n[activation_checkpoint]\nmode = \"selective\"\nselective_ac_option = \"op\"\n\n[checkpoint]\nenable = true\nfolder = \"checkpoint\"\ninterval = 500\n```\n\n**Step 3: Launch training**\n\n```bash\n# 8 GPUs on single node (config selected by name from the registry)\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh\n\n# Override individual fields on the command line\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh --optimizer.lr 3e-4 --training.steps 1000\n\n# Or explicitly with torchrun (run_train.sh wraps this)\ntorchrun --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --module llama3 --config llama3_8b\n```\n\n**Step 4: Monitor and checkpoint**\n\nTensorBoard logs are saved to `.\u002Foutputs\u002Ftb\u002F`:\n```bash\ntensorboard --logdir .\u002Foutputs\u002Ftb\n```\n\n### Workflow 2: Multi-node training with SLURM\n\n```\nMulti-Node Training:\n- [ ] Step 1: Configure parallelism for scale\n- [ ] Step 2: Set up SLURM script\n- [ ] Step 3: Submit job\n- [ ] Step 4: Resume from checkpoint\n```\n\n**Step 1: Configure parallelism for scale**\n\nFor 70B model on 256 GPUs (32 nodes):\n```toml\n[parallelism]\ndata_parallel_shard_degree = 32  # FSDP across 32 ranks\ntensor_parallel_degree = 8        # TP within node\npipeline_parallel_degree = 1      # No PP for 70B\ncontext_parallel_degree = 1       # Increase for long sequences\n```\n\n**Step 2: Set up SLURM script**\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=llama70b\n#SBATCH --nodes=32\n#SBATCH --ntasks-per-node=8\n#SBATCH --gpus-per-node=8\n\nsrun torchrun \\\n  --nnodes=32 \\\n  --nproc_per_node=8 \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  -m torchtitan.train \\\n  --module llama3 --config llama3_70b\n```\n\n**Step 3: Submit job**\n\n```bash\nsbatch multinode_trainer.slurm\n```\n\n**Step 4: Resume from checkpoint**\n\nTraining auto-resumes if checkpoint exists in configured folder.\n\n### Workflow 3: Enable Float8 training for H100s\n\nFloat8 provides 30-50% speedup on H100 GPUs.\n\n```\nFloat8 Training:\n- [ ] Step 1: Install torchao\n- [ ] Step 2: Configure Float8\n- [ ] Step 3: Launch with compile\n```\n\n**Step 1: Install torchao**\n\n```bash\nUSE_CPP=0 pip install git+https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao.git\n```\n\n**Step 2: Configure Float8**\n\nIn the current torchtitan, Float8 is applied at config time via the `quantization`\nparameter in your `model_registry()` call inside the config registry (not via a\n`[quantize.linear.float8]` TOML section). Add a `Float8LinearConverter.Config`:\n\n```python\n# in torchtitan\u002Fmodels\u002Fllama3\u002Fconfig_registry.py (your model_registry(...) call)\nfrom torchtitan.components.quantization import Float8LinearConverter\n\nmodel_spec = model_registry(\n    \"8B\",\n    quantization=[\n        Float8LinearConverter.Config(\n            recipe_name=\"rowwise\",          # or \"rowwise_with_gw_hp\"\n            filter_fqns=[\"output\"],          # skip layers too small to benefit\n            model_compile_enabled=True,      # requires torch.compile for competitive perf\n        ),\n    ],\n)\n```\n\nEnable `torch.compile` in your run config too:\n```toml\n[compile]\nenable = true\ncomponents = [\"model\", \"loss\"]\n```\n\n**Step 3: Launch with compile**\n\n```bash\n# Float8 config is baked into the registered config; just select it and enable compile\nMODULE=llama3 CONFIG=llama3_8b .\u002Frun_train.sh --compile.enable\n```\n\n### Workflow 4: 4D parallelism for 405B models\n\n```\n4D Parallelism (FSDP + TP + PP + CP):\n- [ ] Step 1: Create seed checkpoint\n- [ ] Step 2: Configure 4D parallelism\n- [ ] Step 3: Launch on 512 GPUs\n```\n\n**Step 1: Create seed checkpoint**\n\nRequired for consistent initialization across PP stages:\n```bash\nNGPU=1 MODULE=llama3 CONFIG=llama3_405b .\u002Frun_train.sh \\\n  --checkpoint.enable \\\n  --checkpoint.create_seed_checkpoint \\\n  --parallelism.data_parallel_shard_degree 1 \\\n  --parallelism.tensor_parallel_degree 1 \\\n  --parallelism.pipeline_parallel_degree 1\n```\n\n**Step 2: Configure 4D parallelism**\n\n```toml\n[parallelism]\ndata_parallel_shard_degree = 8   # FSDP\ntensor_parallel_degree = 8       # TP within node\npipeline_parallel_degree = 8     # PP across nodes\ncontext_parallel_degree = 1      # CP for long sequences\n\n[training]\nlocal_batch_size = 32\nseq_len = 8192\n```\n\n**Step 3: Launch on 512 GPUs**\n\n```bash\n# 64 nodes x 8 GPUs = 512 GPUs\nsrun torchrun --nnodes=64 --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --module llama3 --config llama3_405b\n```\n\n## When to use vs alternatives\n\n**Use TorchTitan when:**\n- Pretraining LLMs from scratch (8B to 405B+)\n- Need PyTorch-native solution without third-party dependencies\n- Require composable 4D parallelism (FSDP2, TP, PP, CP)\n- Training on H100s with Float8 support\n- Want interoperable checkpoints with torchtune\u002FHuggingFace\n\n**Use alternatives instead:**\n- **Megatron-LM**: Maximum performance for NVIDIA-only deployments\n- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support\n- **Axolotl\u002FTRL**: Fine-tuning rather than pretraining\n- **LitGPT**: Educational, smaller-scale training\n\n## Common issues\n\n**Issue: Out of memory on large models**\n\nEnable activation checkpointing and reduce batch size:\n```toml\n[activation_checkpoint]\nmode = \"full\"  # Instead of \"selective\"\n\n[training]\nlocal_batch_size = 1\n```\n\nOr use gradient accumulation:\n```toml\n[training]\nlocal_batch_size = 1\nglobal_batch_size = 32  # Accumulates gradients\n```\n\n**Issue: TP causes high memory with async collectives**\n\nSet environment variable:\n```bash\nexport TORCH_NCCL_AVOID_RECORD_STREAMS=1\n```\n\n**Issue: Float8 training not faster**\n\nFloat8 only benefits large GEMMs. Filter small layers via the converter's `filter_fqns`:\n```python\nfrom torchtitan.components.quantization import Float8LinearConverter\n\nFloat8LinearConverter.Config(\n    # add \"auto_filter_small_kn\" to auto-skip layers too small to benefit\n    filter_fqns=[\"attention.wk\", \"attention.wv\", \"output\", \"auto_filter_small_kn\"],\n    model_compile_enabled=True,\n)\n```\n\n**Issue: Checkpoint loading fails after parallelism change**\n\nUse DCP's resharding capability:\n```bash\n# Convert sharded checkpoint to single file\npython -m torch.distributed.checkpoint.format_utils \\\n  dcp_to_torch checkpoint\u002Fstep-1000 checkpoint.pt\n```\n\n**Issue: Pipeline parallelism initialization**\n\nCreate seed checkpoint first (see Workflow 4, Step 1).\n\n## Supported models\n\n| Model | Sizes | Status |\n|-------|-------|--------|\n| Llama 3.1 | 8B, 70B, 405B | Production |\n| Llama 4 | Various | Experimental |\n| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |\n| GPT-OSS | 20B, 120B (MoE) | Experimental |\n| Qwen 3 | Various | Experimental |\n| Flux | Diffusion | Experimental |\n\n## Performance benchmarks (H100)\n\n| Model | GPUs | Parallelism | TPS\u002FGPU | Techniques |\n|-------|------|-------------|---------|------------|\n| Llama 8B | 8 | FSDP | 5,762 | Baseline |\n| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |\n| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |\n| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |\n\n## Advanced topics\n\n**FSDP2 configuration**: See [references\u002Ffsdp.md](references\u002Ffsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.\n\n**Float8 training**: See [references\u002Ffloat8.md](references\u002Ffloat8.md) for tensorwise vs rowwise scaling recipes.\n\n**Checkpointing**: See [references\u002Fcheckpoint.md](references\u002Fcheckpoint.md) for HuggingFace conversion and async checkpointing.\n\n**Adding custom models**: See [references\u002Fcustom-models.md](references\u002Fcustom-models.md) for TrainSpec protocol.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.06511\n- ICLR 2025: https:\u002F\u002Ficlr.cc\u002Fvirtual\u002F2025\u002Fposter\u002F29620\n- PyTorch Forum: https:\u002F\u002Fdiscuss.pytorch.org\u002Fc\u002Fdistributed\u002Ftorchtitan\u002F44\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":39,"description":7,"dependencies":42},"Orchestra Research","1.0.1",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38],"Model Architecture","Distributed Training","TorchTitan","FSDP2","Tensor Parallel","Pipeline Parallel","Context Parallel","Float8","Llama","Pretraining",[40,41],"linux","macos",[43,44,45],"torch>=2.6.0","torchtitan>=0.2.0","torchao>=0.5.0","main","2e6aa8d15ab0a366c90be54f61bf9c6e97015e83","skills\u002Fmlops\u002Ftorchtitan",1,"https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet","steadfaste.com","https:\u002F\u002Fgithub.com\u002Flukemcqueen","AI Architect | Building Hermes Cortex: governed multi-agent AI for autonomous software engineering | Enterprise Tech Consultant","2026-11-14 22:33:34",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add lukemcqueen\u002Fsteadfaste-fleet --skill distributed-llm-pretraining-torchtitan",{"tool":61,"command":62},"bunx","bunx skills add lukemcqueen\u002Fsteadfaste-fleet --skill distributed-llm-pretraining-torchtitan",{"tool":64,"command":65},"pnpm","pnpm dlx skills add lukemcqueen\u002Fsteadfaste-fleet --skill distributed-llm-pretraining-torchtitan","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fmlops\u002Ftorchtitan\u002FSKILL.md",true,[],[70,78,85,92,99,106],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":77},"01M7DXY4G8NSVB59K8B6S2AQNP","macbook-linux-maintenance","Legacy MacBook Linux fixes: GPU wedge, reboot hang, thermal.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fmacbook-linux-maintenance\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":77},"01M7DXXYJCXFJ3TGB84ZQ8KFE6","governance-regression-triage","Triage a host's governance issues: what changed.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fgovernance-regression-triage\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":77},"01M7DXXSAP1G8FDRJAMXQGBBMK","enforcement-immutability","Use when enforcement files lose their immutable flag.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fenforcement-immutability\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":77},"01M78SSDQVRYX64Y16X32TC0NS","steadfaste-fleet-repo-deploy","Use when committing to the steadfaste-fleet repo.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fsteadfaste-fleet-repo-deploy\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":77},"01M78SS928F3R2RJAYVW4E30CE","steadfaste-fleet-maintenance","Maintain an installed Steadfaste Fleet instance — update both the upstream Hermes Agent and the fleet repo layer, merge diverged histories, sync skills safely, and troubleshoot common issues. Covers the dual-update workflow (hermes update + fleet-update.sh), encrypted brain data backup, mycortex import, and recovery techniques.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fsteadfaste-fleet-maintenance\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":77},"01M78SP1CHW0T0BWE5ZT6JBPX8","fleet-preflight","DEPRECATED alias — merged into survey-before-action (2026-08-20). Load survey-before-action instead; its \"Repo-Specific Pre-Flight\" section covers git search, Hermes boundary, and deployment verification.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Ffleet-preflight\u002FSKILL.md",[],[]]