[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QM5VGDFTSZ9Z144TV3VMG7":3,"skill-same:01M6QM5VGDFTSZ9Z144TV3VMG7":99},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":58,"commitSha":59,"skillDirectory":60,"forksCount":14,"repositoryHtmlUrl":61,"repositoryHomepageUrl":62,"ownerHtmlUrl":63,"ownerBlogUrl":64,"ownerDescription":65,"updatedAt":66,"install":67,"resources":80},"01M6QM5VGDFTSZ9Z144TV3VMG7","distributed-llm-pretraining-torchtitan",null,"Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-torchtitan\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"distributed","Pretraining","torchtitan","distributed llm pretraining torchtitan","compile","Distributed Training","pipeline","model","parallel","context","training","Model Architecture","FSDP2","pytorch","architecture","native","pp","at","orchestra","checkpointing","2026-11-10 10:12:30",false,"---\nname: distributed-llm-pretraining-torchtitan\ndescription: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Model Architecture, Distributed Training, TorchTitan, FSDP2, Tensor Parallel, Pipeline Parallel, Context Parallel, Float8, Llama, Pretraining]\ndependencies: [torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0]\ncategory: \"ai-research\"\n---\n\n# TorchTitan - PyTorch Native Distributed LLM Pretraining\n\n## Quick start\n\nTorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.\n\n**Installation**:\n```bash\n# From PyPI (stable)\npip install torchtitan\n\n# From source (latest features, requires PyTorch nightly)\ngit clone https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\ncd torchtitan\npip install -r requirements.txt\n```\n\n**Download tokenizer**:\n```bash\n# Get HF token from https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\npython scripts\u002Fdownload_hf_assets.py --repo_id meta-llama\u002FLlama-3.1-8B --assets tokenizer --hf_token=...\n```\n\n**Start training on 8 GPUs**:\n```bash\nCONFIG_FILE=\".\u002Ftorchtitan\u002Fmodels\u002Fllama3\u002Ftrain_configs\u002Fllama3_8b.toml\" .\u002Frun_train.sh\n```\n\n## Common workflows\n\n### Workflow 1: Pretrain Llama 3.1 8B on single node\n\nCopy this checklist:\n\n```\nSingle Node Pretraining:\n- [ ] Step 1: Download tokenizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Launch training\n- [ ] Step 4: Monitor and checkpoint\n```\n\n**Step 1: Download tokenizer**\n\n```bash\npython scripts\u002Fdownload_hf_assets.py \\\n  --repo_id meta-llama\u002FLlama-3.1-8B \\\n  --assets tokenizer \\\n  --hf_token=YOUR_HF_TOKEN\n```\n\n**Step 2: Configure training**\n\nEdit or create a TOML config file:\n\n```toml\n# llama3_8b_custom.toml\n[job]\ndump_folder = \".\u002Foutputs\"\ndescription = \"Llama 3.1 8B training\"\n\n[model]\nname = \"llama3\"\nflavor = \"8B\"\nhf_assets_path = \".\u002Fassets\u002Fhf\u002FLlama-3.1-8B\"\n\n[optimizer]\nname = \"AdamW\"\nlr = 3e-4\n\n[lr_scheduler]\nwarmup_steps = 200\n\n[training]\nlocal_batch_size = 2\nseq_len = 8192\nmax_norm = 1.0\nsteps = 1000\ndataset = \"c4\"\n\n[parallelism]\ndata_parallel_shard_degree = -1  # Use all GPUs for FSDP\n\n[activation_checkpoint]\nmode = \"selective\"\nselective_ac_option = \"op\"\n\n[checkpoint]\nenable = true\nfolder = \"checkpoint\"\ninterval = 500\n```\n\n**Step 3: Launch training**\n\n```bash\n# 8 GPUs on single node\nCONFIG_FILE=\".\u002Fllama3_8b_custom.toml\" .\u002Frun_train.sh\n\n# Or explicitly with torchrun\ntorchrun --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_8b_custom.toml\n```\n\n**Step 4: Monitor and checkpoint**\n\nTensorBoard logs are saved to `.\u002Foutputs\u002Ftb\u002F`:\n```bash\ntensorboard --logdir .\u002Foutputs\u002Ftb\n```\n\n### Workflow 2: Multi-node training with SLURM\n\n```\nMulti-Node Training:\n- [ ] Step 1: Configure parallelism for scale\n- [ ] Step 2: Set up SLURM script\n- [ ] Step 3: Submit job\n- [ ] Step 4: Resume from checkpoint\n```\n\n**Step 1: Configure parallelism for scale**\n\nFor 70B model on 256 GPUs (32 nodes):\n```toml\n[parallelism]\ndata_parallel_shard_degree = 32  # FSDP across 32 ranks\ntensor_parallel_degree = 8        # TP within node\npipeline_parallel_degree = 1      # No PP for 70B\ncontext_parallel_degree = 1       # Increase for long sequences\n```\n\n**Step 2: Set up SLURM script**\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=llama70b\n#SBATCH --nodes=32\n#SBATCH --ntasks-per-node=8\n#SBATCH --gpus-per-node=8\n\nsrun torchrun \\\n  --nnodes=32 \\\n  --nproc_per_node=8 \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_70b.toml\n```\n\n**Step 3: Submit job**\n\n```bash\nsbatch multinode_trainer.slurm\n```\n\n**Step 4: Resume from checkpoint**\n\nTraining auto-resumes if checkpoint exists in configured folder.\n\n### Workflow 3: Enable Float8 training for H100s\n\nFloat8 provides 30-50% speedup on H100 GPUs.\n\n```\nFloat8 Training:\n- [ ] Step 1: Install torchao\n- [ ] Step 2: Configure Float8\n- [ ] Step 3: Launch with compile\n```\n\n**Step 1: Install torchao**\n\n```bash\nUSE_CPP=0 pip install git+https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao.git\n```\n\n**Step 2: Configure Float8**\n\nAdd to your TOML config:\n```toml\n[model]\nconverters = [\"quantize.linear.float8\"]\n\n[quantize.linear.float8]\nenable_fsdp_float8_all_gather = true\nprecompute_float8_dynamic_scale_for_fsdp = true\nfilter_fqns = [\"output\"]  # Exclude output layer\n\n[compile]\nenable = true\ncomponents = [\"model\", \"loss\"]\n```\n\n**Step 3: Launch with compile**\n\n```bash\nCONFIG_FILE=\".\u002Fllama3_8b.toml\" .\u002Frun_train.sh \\\n  --model.converters=\"quantize.linear.float8\" \\\n  --quantize.linear.float8.enable_fsdp_float8_all_gather \\\n  --compile.enable\n```\n\n### Workflow 4: 4D parallelism for 405B models\n\n```\n4D Parallelism (FSDP + TP + PP + CP):\n- [ ] Step 1: Create seed checkpoint\n- [ ] Step 2: Configure 4D parallelism\n- [ ] Step 3: Launch on 512 GPUs\n```\n\n**Step 1: Create seed checkpoint**\n\nRequired for consistent initialization across PP stages:\n```bash\nNGPU=1 CONFIG_FILE=.\u002Fllama3_405b.toml .\u002Frun_train.sh \\\n  --checkpoint.enable \\\n  --checkpoint.create_seed_checkpoint \\\n  --parallelism.data_parallel_shard_degree 1 \\\n  --parallelism.tensor_parallel_degree 1 \\\n  --parallelism.pipeline_parallel_degree 1\n```\n\n**Step 2: Configure 4D parallelism**\n\n```toml\n[parallelism]\ndata_parallel_shard_degree = 8   # FSDP\ntensor_parallel_degree = 8       # TP within node\npipeline_parallel_degree = 8     # PP across nodes\ncontext_parallel_degree = 1      # CP for long sequences\n\n[training]\nlocal_batch_size = 32\nseq_len = 8192\n```\n\n**Step 3: Launch on 512 GPUs**\n\n```bash\n# 64 nodes x 8 GPUs = 512 GPUs\nsrun torchrun --nnodes=64 --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_405b.toml\n```\n\n## When to use vs alternatives\n\n**Use TorchTitan when:**\n- Pretraining LLMs from scratch (8B to 405B+)\n- Need PyTorch-native solution without third-party dependencies\n- Require composable 4D parallelism (FSDP2, TP, PP, CP)\n- Training on H100s with Float8 support\n- Want interoperable checkpoints with torchtune\u002FHuggingFace\n\n**Use alternatives instead:**\n- **Megatron-LM**: Maximum performance for NVIDIA-only deployments\n- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support\n- **Axolotl\u002FTRL**: Fine-tuning rather than pretraining\n- **LitGPT**: Educational, smaller-scale training\n\n## Common issues\n\n**Issue: Out of memory on large models**\n\nEnable activation checkpointing and reduce batch size:\n```toml\n[activation_checkpoint]\nmode = \"full\"  # Instead of \"selective\"\n\n[training]\nlocal_batch_size = 1\n```\n\nOr use gradient accumulation:\n```toml\n[training]\nlocal_batch_size = 1\nglobal_batch_size = 32  # Accumulates gradients\n```\n\n**Issue: TP causes high memory with async collectives**\n\nSet environment variable:\n```bash\nexport TORCH_NCCL_AVOID_RECORD_STREAMS=1\n```\n\n**Issue: Float8 training not faster**\n\nFloat8 only benefits large GEMMs. Filter small layers:\n```toml\n[quantize.linear.float8]\nfilter_fqns = [\"attention.wk\", \"attention.wv\", \"output\", \"auto_filter_small_kn\"]\n```\n\n**Issue: Checkpoint loading fails after parallelism change**\n\nUse DCP's resharding capability:\n```bash\n# Convert sharded checkpoint to single file\npython -m torch.distributed.checkpoint.format_utils \\\n  dcp_to_torch checkpoint\u002Fstep-1000 checkpoint.pt\n```\n\n**Issue: Pipeline parallelism initialization**\n\nCreate seed checkpoint first (see Workflow 4, Step 1).\n\n## Supported models\n\n| Model | Sizes | Status |\n|-------|-------|--------|\n| Llama 3.1 | 8B, 70B, 405B | Production |\n| Llama 4 | Various | Experimental |\n| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |\n| GPT-OSS | 20B, 120B (MoE) | Experimental |\n| Qwen 3 | Various | Experimental |\n| Flux | Diffusion | Experimental |\n\n## Performance benchmarks (H100)\n\n| Model | GPUs | Parallelism | TPS\u002FGPU | Techniques |\n|-------|------|-------------|---------|------------|\n| Llama 8B | 8 | FSDP | 5,762 | Baseline |\n| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |\n| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |\n| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |\n\n## Advanced topics\n\n**FSDP2 configuration**: See [references\u002Ffsdp.md](references\u002Ffsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.\n\n**Float8 training**: See [references\u002Ffloat8.md](references\u002Ffloat8.md) for tensorwise vs rowwise scaling recipes.\n\n**Checkpointing**: See [references\u002Fcheckpoint.md](references\u002Fcheckpoint.md) for HuggingFace conversion and async checkpointing.\n\n**Adding custom models**: See [references\u002Fcustom-models.md](references\u002Fcustom-models.md) for TrainSpec protocol.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.06511\n- ICLR 2025: https:\u002F\u002Ficlr.cc\u002Fvirtual\u002F2025\u002Fposter\u002F29620\n- PyTorch Forum: https:\u002F\u002Fdiscuss.pytorch.org\u002Fc\u002Fdistributed\u002Ftorchtitan\u002F44\n\n","# TorchTitan - PyTorch Native Distributed LLM Pretraining\n\n## Quick start\n\nTorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.\n\n**Installation**:\n```bash\n# From PyPI (stable)\npip install torchtitan\n\n# From source (latest features, requires PyTorch nightly)\ngit clone https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\ncd torchtitan\npip install -r requirements.txt\n```\n\n**Download tokenizer**:\n```bash\n# Get HF token from https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\npython scripts\u002Fdownload_hf_assets.py --repo_id meta-llama\u002FLlama-3.1-8B --assets tokenizer --hf_token=...\n```\n\n**Start training on 8 GPUs**:\n```bash\nCONFIG_FILE=\".\u002Ftorchtitan\u002Fmodels\u002Fllama3\u002Ftrain_configs\u002Fllama3_8b.toml\" .\u002Frun_train.sh\n```\n\n## Common workflows\n\n### Workflow 1: Pretrain Llama 3.1 8B on single node\n\nCopy this checklist:\n\n```\nSingle Node Pretraining:\n- [ ] Step 1: Download tokenizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Launch training\n- [ ] Step 4: Monitor and checkpoint\n```\n\n**Step 1: Download tokenizer**\n\n```bash\npython scripts\u002Fdownload_hf_assets.py \\\n  --repo_id meta-llama\u002FLlama-3.1-8B \\\n  --assets tokenizer \\\n  --hf_token=YOUR_HF_TOKEN\n```\n\n**Step 2: Configure training**\n\nEdit or create a TOML config file:\n\n```toml\n# llama3_8b_custom.toml\n[job]\ndump_folder = \".\u002Foutputs\"\ndescription = \"Llama 3.1 8B training\"\n\n[model]\nname = \"llama3\"\nflavor = \"8B\"\nhf_assets_path = \".\u002Fassets\u002Fhf\u002FLlama-3.1-8B\"\n\n[optimizer]\nname = \"AdamW\"\nlr = 3e-4\n\n[lr_scheduler]\nwarmup_steps = 200\n\n[training]\nlocal_batch_size = 2\nseq_len = 8192\nmax_norm = 1.0\nsteps = 1000\ndataset = \"c4\"\n\n[parallelism]\ndata_parallel_shard_degree = -1  # Use all GPUs for FSDP\n\n[activation_checkpoint]\nmode = \"selective\"\nselective_ac_option = \"op\"\n\n[checkpoint]\nenable = true\nfolder = \"checkpoint\"\ninterval = 500\n```\n\n**Step 3: Launch training**\n\n```bash\n# 8 GPUs on single node\nCONFIG_FILE=\".\u002Fllama3_8b_custom.toml\" .\u002Frun_train.sh\n\n# Or explicitly with torchrun\ntorchrun --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_8b_custom.toml\n```\n\n**Step 4: Monitor and checkpoint**\n\nTensorBoard logs are saved to `.\u002Foutputs\u002Ftb\u002F`:\n```bash\ntensorboard --logdir .\u002Foutputs\u002Ftb\n```\n\n### Workflow 2: Multi-node training with SLURM\n\n```\nMulti-Node Training:\n- [ ] Step 1: Configure parallelism for scale\n- [ ] Step 2: Set up SLURM script\n- [ ] Step 3: Submit job\n- [ ] Step 4: Resume from checkpoint\n```\n\n**Step 1: Configure parallelism for scale**\n\nFor 70B model on 256 GPUs (32 nodes):\n```toml\n[parallelism]\ndata_parallel_shard_degree = 32  # FSDP across 32 ranks\ntensor_parallel_degree = 8        # TP within node\npipeline_parallel_degree = 1      # No PP for 70B\ncontext_parallel_degree = 1       # Increase for long sequences\n```\n\n**Step 2: Set up SLURM script**\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=llama70b\n#SBATCH --nodes=32\n#SBATCH --ntasks-per-node=8\n#SBATCH --gpus-per-node=8\n\nsrun torchrun \\\n  --nnodes=32 \\\n  --nproc_per_node=8 \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_70b.toml\n```\n\n**Step 3: Submit job**\n\n```bash\nsbatch multinode_trainer.slurm\n```\n\n**Step 4: Resume from checkpoint**\n\nTraining auto-resumes if checkpoint exists in configured folder.\n\n### Workflow 3: Enable Float8 training for H100s\n\nFloat8 provides 30-50% speedup on H100 GPUs.\n\n```\nFloat8 Training:\n- [ ] Step 1: Install torchao\n- [ ] Step 2: Configure Float8\n- [ ] Step 3: Launch with compile\n```\n\n**Step 1: Install torchao**\n\n```bash\nUSE_CPP=0 pip install git+https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao.git\n```\n\n**Step 2: Configure Float8**\n\nAdd to your TOML config:\n```toml\n[model]\nconverters = [\"quantize.linear.float8\"]\n\n[quantize.linear.float8]\nenable_fsdp_float8_all_gather = true\nprecompute_float8_dynamic_scale_for_fsdp = true\nfilter_fqns = [\"output\"]  # Exclude output layer\n\n[compile]\nenable = true\ncomponents = [\"model\", \"loss\"]\n```\n\n**Step 3: Launch with compile**\n\n```bash\nCONFIG_FILE=\".\u002Fllama3_8b.toml\" .\u002Frun_train.sh \\\n  --model.converters=\"quantize.linear.float8\" \\\n  --quantize.linear.float8.enable_fsdp_float8_all_gather \\\n  --compile.enable\n```\n\n### Workflow 4: 4D parallelism for 405B models\n\n```\n4D Parallelism (FSDP + TP + PP + CP):\n- [ ] Step 1: Create seed checkpoint\n- [ ] Step 2: Configure 4D parallelism\n- [ ] Step 3: Launch on 512 GPUs\n```\n\n**Step 1: Create seed checkpoint**\n\nRequired for consistent initialization across PP stages:\n```bash\nNGPU=1 CONFIG_FILE=.\u002Fllama3_405b.toml .\u002Frun_train.sh \\\n  --checkpoint.enable \\\n  --checkpoint.create_seed_checkpoint \\\n  --parallelism.data_parallel_shard_degree 1 \\\n  --parallelism.tensor_parallel_degree 1 \\\n  --parallelism.pipeline_parallel_degree 1\n```\n\n**Step 2: Configure 4D parallelism**\n\n```toml\n[parallelism]\ndata_parallel_shard_degree = 8   # FSDP\ntensor_parallel_degree = 8       # TP within node\npipeline_parallel_degree = 8     # PP across nodes\ncontext_parallel_degree = 1      # CP for long sequences\n\n[training]\nlocal_batch_size = 32\nseq_len = 8192\n```\n\n**Step 3: Launch on 512 GPUs**\n\n```bash\n# 64 nodes x 8 GPUs = 512 GPUs\nsrun torchrun --nnodes=64 --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_405b.toml\n```\n\n## When to use vs alternatives\n\n**Use TorchTitan when:**\n- Pretraining LLMs from scratch (8B to 405B+)\n- Need PyTorch-native solution without third-party dependencies\n- Require composable 4D parallelism (FSDP2, TP, PP, CP)\n- Training on H100s with Float8 support\n- Want interoperable checkpoints with torchtune\u002FHuggingFace\n\n**Use alternatives instead:**\n- **Megatron-LM**: Maximum performance for NVIDIA-only deployments\n- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support\n- **Axolotl\u002FTRL**: Fine-tuning rather than pretraining\n- **LitGPT**: Educational, smaller-scale training\n\n## Common issues\n\n**Issue: Out of memory on large models**\n\nEnable activation checkpointing and reduce batch size:\n```toml\n[activation_checkpoint]\nmode = \"full\"  # Instead of \"selective\"\n\n[training]\nlocal_batch_size = 1\n```\n\nOr use gradient accumulation:\n```toml\n[training]\nlocal_batch_size = 1\nglobal_batch_size = 32  # Accumulates gradients\n```\n\n**Issue: TP causes high memory with async collectives**\n\nSet environment variable:\n```bash\nexport TORCH_NCCL_AVOID_RECORD_STREAMS=1\n```\n\n**Issue: Float8 training not faster**\n\nFloat8 only benefits large GEMMs. Filter small layers:\n```toml\n[quantize.linear.float8]\nfilter_fqns = [\"attention.wk\", \"attention.wv\", \"output\", \"auto_filter_small_kn\"]\n```\n\n**Issue: Checkpoint loading fails after parallelism change**\n\nUse DCP's resharding capability:\n```bash\n# Convert sharded checkpoint to single file\npython -m torch.distributed.checkpoint.format_utils \\\n  dcp_to_torch checkpoint\u002Fstep-1000 checkpoint.pt\n```\n\n**Issue: Pipeline parallelism initialization**\n\nCreate seed checkpoint first (see Workflow 4, Step 1).\n\n## Supported models\n\n| Model | Sizes | Status |\n|-------|-------|--------|\n| Llama 3.1 | 8B, 70B, 405B | Production |\n| Llama 4 | Various | Experimental |\n| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |\n| GPT-OSS | 20B, 120B (MoE) | Experimental |\n| Qwen 3 | Various | Experimental |\n| Flux | Diffusion | Experimental |\n\n## Performance benchmarks (H100)\n\n| Model | GPUs | Parallelism | TPS\u002FGPU | Techniques |\n|-------|------|-------------|---------|------------|\n| Llama 8B | 8 | FSDP | 5,762 | Baseline |\n| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |\n| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |\n| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |\n\n## Advanced topics\n\n**FSDP2 configuration**: See [references\u002Ffsdp.md](references\u002Ffsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.\n\n**Float8 training**: See [references\u002Ffloat8.md](references\u002Ffloat8.md) for tensorwise vs rowwise scaling recipes.\n\n**Checkpointing**: See [references\u002Fcheckpoint.md](references\u002Fcheckpoint.md) for HuggingFace conversion and async checkpointing.\n\n**Adding custom models**: See [references\u002Fcustom-models.md](references\u002Fcustom-models.md) for TrainSpec protocol.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.06511\n- ICLR 2025: https:\u002F\u002Ficlr.cc\u002Fvirtual\u002F2025\u002Fposter\u002F29620\n- PyTorch Forum: https:\u002F\u002Fdiscuss.pytorch.org\u002Fc\u002Fdistributed\u002Ftorchtitan\u002F44\n\n",{"name":5,"tags":44,"author":51,"license":15,"version":52,"category":53,"description":7,"dependencies":54},[30,24,45,31,46,47,48,49,50,20],"TorchTitan","Tensor Parallel","Pipeline Parallel","Context Parallel","Float8","Llama","Orchestra Research","1.0.0","ai-research",[55,56,57],"torch>=2.6.0","torchtitan>=0.2.0","torchao>=0.5.0","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Forch-torchtitan","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":68,"prompt":78,"zipAvailable":79},[69,72,75],{"tool":70,"command":71},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill distributed-llm-pretraining-torchtitan",{"tool":73,"command":74},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill distributed-llm-pretraining-torchtitan",{"tool":76,"command":77},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill distributed-llm-pretraining-torchtitan","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-torchtitan\u002FSKILL.md",true,[81,87,91,95],{"path":82,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fcheckpoint.md","REFERENCE","参考文档",4216,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-torchtitan\u002Freferences\u002Fcheckpoint.md",{"path":88,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fcustom-models.md",7383,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-torchtitan\u002Freferences\u002Fcustom-models.md",{"path":92,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Ffloat8.md",4055,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-torchtitan\u002Freferences\u002Ffloat8.md",{"path":96,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":97,"sourceHtmlUrl":98},"references\u002Ffsdp.md",3888,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-torchtitan\u002Freferences\u002Ffsdp.md",[100,132,157,184,209,231],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":111,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[106],{"categoryNo":107,"categoryName":108,"taxonomyType":109,"taxonomyTypeDesc":110},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","for","browsing","request","youtube transcripts",{"skillNo":133,"name":134,"nameZh":6,"description":135,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":136,"categories":137,"keywords":138,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[139,140,141,142,116,143,144,145,146,147,148,149,150,151,152,153,154,155,156,128],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","use","platform","grow","posting","competitive",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":166,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[163],{"categoryNo":164,"categoryName":165,"taxonomyType":109,"taxonomyTypeDesc":110},"01JBSEEDCAT000000000000023","Skill与智能体开发",[167,150,168,169,170,171,172,173,152,174,175,58,176,177,178,179,180,181,182,183],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":185,"name":186,"nameZh":6,"description":187,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":188,"categories":189,"keywords":190,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[191,192,193,194,195,196,197,198,199,152,200,123,201,202,203,204,205,206,207,208],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":210,"name":211,"nameZh":6,"description":212,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":213,"categories":214,"keywords":216,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[215],{"categoryNo":164,"categoryName":165,"taxonomyType":109,"taxonomyTypeDesc":110},[217,218,219,220,221,169,196,222,223,25,224,225,150,28,122,226,227,228,229,230],"workflow","workflow builder","builder","design","workflows","code","orchestrate","loop","multi-agent","across","fresh","scripts","judge","claude code workflow",{"skillNo":232,"name":233,"nameZh":6,"description":234,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":235,"categories":236,"keywords":237,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[238,239,240,241,242,243,244,245,246,224,247,248,249,250,251,252,124,253,254,208],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]