[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QKYER1XNSRGJD04VT3S8EX":3,"skill-same:01M6QKYER1XNSRGJD04VT3S8EX":132},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":44,"linkOnly":45,"markdownContent":46,"markdownBody":47,"frontMatter":48,"refName":59,"commitSha":60,"skillDirectory":61,"forksCount":14,"repositoryHtmlUrl":62,"repositoryHomepageUrl":63,"ownerHtmlUrl":64,"ownerBlogUrl":65,"ownerDescription":66,"updatedAt":67,"install":68,"resources":81},"01M6QKYER1XNSRGJD04VT3S8EX","pytorch-fsdp2",null,"Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision\u002Foffload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],"pytorch fsdp2","pytorch","FSDP2","use","you","Torch Distributed","Sharded Checkpointing","Device Mesh","DTensor","Fully Sharded Data Parallel","models","mesh","parallel","shard","training","distributed","config","init","research","memory","2026-11-10 10:12:30",false,"---\nname: pytorch-fsdp2\ndescription: Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision\u002Foffload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [PyTorch, FSDP2, Fully Sharded Data Parallel, Distributed Training, DTensor, Device Mesh, Sharded Checkpointing, Mixed Precision, Offload, Torch Distributed]\ndependencies: [torch]\ncategory: \"ai-research\"\n---\n\n# Skill: Use PyTorch FSDP2 (`fully_shard`) correctly in a training script\n\nThis skill teaches a coding agent how to **add PyTorch FSDP2** to a training loop with correct initialization, sharding, mixed precision\u002Foffload configuration, and checkpointing.\n\n> FSDP2 in PyTorch is exposed primarily via `torch.distributed.fsdp.fully_shard` and the `FSDPModule` methods it adds in-place to modules. See: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## When to use this skill\n\nUse FSDP2 when:\n- Your model **doesn’t fit** on one GPU (parameters + gradients + optimizer state).\n- You want an eager-mode sharding approach that is **DTensor-based per-parameter sharding** (more inspectable, simpler sharded state dicts) than FSDP1.  \n- You may later compose DP with **Tensor Parallel** using **DeviceMesh**.\n\nAvoid (or be careful) if:\n- You need strict backwards-compatible checkpoints across PyTorch versions (DCP warns against this).\n- You’re forced onto older PyTorch versions without the FSDP2 stack.\n\n## Alternatives (when FSDP2 is not the best fit)\n\n- **DistributedDataParallel (DDP)**: Use the standard data-parallel wrapper when you want classic distributed data parallel training.\n- **FullyShardedDataParallel (FSDP1)**: Use the original FSDP wrapper for parameter sharding across data-parallel workers.\n\nReference: `references\u002Fpytorch_ddp_notes.md`, `references\u002Fpytorch_fsdp1_api.md`.\n\n---\n\n## Contract the agent must follow\n\n1. **Launch with `torchrun`** and set the CUDA device per process (usually via `LOCAL_RANK`).  \n2. **Apply `fully_shard()` bottom-up**, i.e., shard submodules (e.g., Transformer blocks) before the root module.  \n3. **Call `model(input)`**, not `model.forward(input)`, so the FSDP2 hooks run (unless you explicitly `unshard()` or register the forward method).  \n4. **Create the optimizer after sharding** and make sure it is built on the **DTensor parameters** (post-`fully_shard`).  \n5. **Checkpoint using Distributed Checkpoint (DCP)** or the distributed-state-dict helpers, not naïve `torch.save(model.state_dict())` unless you deliberately gather to full tensors.\n\n(Each of these rules is directly described in the official API docs\u002Ftutorial; see references.)\n\n---\n\n## Step-by-step procedure\n\n### 0) Version & environment sanity\n- Prefer a recent stable PyTorch where the docs show FSDP2 and DCP updated recently.\n- Use `torchrun --nproc_per_node \u003Cgpus_per_node> ...` and ensure `RANK`, `WORLD_SIZE`, `LOCAL_RANK` are visible.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (launch commands and setup), `references\u002Fpytorch_fully_shard_api.md` (user contract).\n\n---\n\n### 1) Initialize distributed and set device\nMinimal, correct pattern:\n- `dist.init_process_group(backend=\"nccl\")`\n- `torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))`\n- Optionally create a `DeviceMesh` to describe the data-parallel group(s)\n\nReference: `references\u002Fpytorch_device_mesh_tutorial.md` (why DeviceMesh exists & how it manages process groups).\n\n---\n\n### 2) Build model on meta device (recommended for very large models)\nFor big models, initialize on `meta`, apply sharding, then materialize weights on GPU:\n- `with torch.device(\"meta\"): model = ...`\n- apply `fully_shard(...)` on submodules, then `fully_shard(model)`\n- `model.to_empty(device=\"cuda\")`\n- `model.reset_parameters()` (or your init routine)\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (migration guide shows this flow explicitly).\n\n---\n\n### 3) Apply `fully_shard()` bottom-up (wrapping policy = “apply where needed”)\n**Do not** only call `fully_shard` on the topmost module.\n\nRecommended sharding pattern for transformer-like models:\n- iterate modules, `if isinstance(m, TransformerBlock): fully_shard(m, ...)`\n- then `fully_shard(model, ...)`\n\nWhy:\n- `fully_shard` forms “parameter groups” for collective efficiency and excludes params already grouped by earlier calls. Bottom-up gives better overlap and lower peak memory.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (bottom-up requirement and why).\n\n---\n\n### 4) Configure `reshard_after_forward` for memory\u002Fperf trade-offs\nDefault behavior:\n- `None` means `True` for non-root modules and `False` for root modules (good default).\n\nHeuristics:\n- If you’re memory-bound: keep defaults or force `True` on many blocks.\n- If you’re throughput-bound and can afford memory: consider keeping unsharded params longer (root often `False`).\n- Advanced: use an `int` to reshard to a smaller mesh after forward (e.g., intra-node) if it’s a meaningful divisor.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (full semantics).\n\n---\n\n### 5) Mixed precision & offload (optional but common)\nFSDP2 uses:\n- `mp_policy=MixedPrecisionPolicy(param_dtype=..., reduce_dtype=..., output_dtype=..., cast_forward_inputs=...)`\n- `offload_policy=CPUOffloadPolicy()` if you want CPU offload\n\nRules of thumb:\n- Start with BF16 parameters\u002Freductions on H100\u002FA100-class GPUs (if numerically stable for your model).\n- Keep `reduce_dtype` aligned with your gradient reduction expectations.\n- If you use CPU offload, budget for PCIe\u002FNVLink traffic and runtime overhead.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (MixedPrecisionPolicy \u002F OffloadPolicy classes).\n\n---\n\n### 6) Optimizer, gradient clipping, accumulation\n- Create the optimizer **after** sharding so it holds DTensor params.\n- If you need gradient accumulation \u002F no_sync:\n  - use the FSDP2 mechanism (`set_requires_gradient_sync`) instead of FSDP1’s `no_sync()`.\n\nGradient clipping:\n- Use the approach shown in the FSDP2 tutorial (“Gradient Clipping and Optimizer with DTensor”), because parameters\u002Fgradients are DTensors.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n### 7) Checkpointing: prefer DCP or distributed state dict helpers\nTwo recommended approaches:\n\n**A) Distributed Checkpoint (DCP) — best default**\n- DCP saves\u002Floads from multiple ranks in parallel and supports load-time resharding.\n- DCP produces **multiple files** (often at least one per rank) and operates “in place”.\n\n**B) Distributed state dict helpers**\n- `get_model_state_dict` \u002F `set_model_state_dict` with `StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True, ...)`\n- For optimizer: `get_optimizer_state_dict` \u002F `set_optimizer_state_dict`\n\nAvoid:\n- Saving DTensor state dicts with plain `torch.save` unless you intentionally convert with `DTensor.full_tensor()` and manage memory carefully.\n\nReferences:\n- `references\u002Fpytorch_dcp_overview.md` (DCP behavior and caveats)\n- `references\u002Fpytorch_dcp_recipe.md` and `references\u002Fpytorch_dcp_async_recipe.md` (end-to-end usage)\n- `references\u002Fpytorch_fsdp2_tutorial.md` (DTensor vs DCP state-dict flows)\n- `references\u002Fpytorch_examples_fsdp2.md` (working checkpoint scripts)\n\n---\n\n## Workflow checklists (copy-paste friendly)\n\n### Workflow A: Retrofit FSDP2 into an existing training script\n- [ ] Launch with `torchrun` and initialize the process group.\n- [ ] Set the CUDA device from `LOCAL_RANK`; create a `DeviceMesh` if you need multi-dim parallelism.\n- [ ] Build the model (use `meta` if needed), apply `fully_shard` bottom-up, then `fully_shard(model)`.\n- [ ] Create the optimizer after sharding so it captures DTensor parameters.\n- [ ] Use `model(inputs)` so hooks run; use `set_requires_gradient_sync` for accumulation.\n- [ ] Add DCP save\u002Fload via `torch.distributed.checkpoint` helpers.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`, `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_device_mesh_tutorial.md`, `references\u002Fpytorch_dcp_recipe.md`.\n\n### Workflow B: Add DCP save\u002Fload (minimal pattern)\n- [ ] Wrap state in `Stateful` or assemble state via `get_state_dict`.\n- [ ] Call `dcp.save(...)` from all ranks to a shared path.\n- [ ] Call `dcp.load(...)` and restore with `set_state_dict`.\n- [ ] Validate any resharding assumptions when loading into a different mesh.\n\nReference: `references\u002Fpytorch_dcp_recipe.md`.\n\n## Debug checklist (what the agent should check first)\n\n1. **All ranks on distinct GPUs?**  \n   If not, verify `torch.cuda.set_device(LOCAL_RANK)` and your `torchrun` flags.\n2. **Did you accidentally call `forward()` directly?**  \n   Use `model(input)` or explicitly `unshard()` \u002F register forward.\n3. **Is `fully_shard()` applied bottom-up?**  \n   If only root is sharded, expect worse memory\u002Fperf and possible confusion.\n4. **Optimizer created at the right time?**  \n   Must be built on DTensor parameters *after* sharding.\n5. **Checkpointing path consistent?**  \n   - If using DCP, don’t mix with ad-hoc `torch.save` unless you understand conversions.\n   - Be mindful of PyTorch-version compatibility warnings for DCP.\n\n---\n\n## Common issues and fixes\n\n- **Forward hooks not running** → Call `model(inputs)` (or `unshard()` explicitly) instead of `model.forward(...)`.\n- **Optimizer sees non-DTensor params** → Create optimizer after all `fully_shard` calls.\n- **Only root module sharded** → Apply `fully_shard` bottom-up on submodules before the root.\n- **Memory spikes after forward** → Set `reshard_after_forward=True` for more modules.\n- **Gradient accumulation desync** → Use `set_requires_gradient_sync` instead of FSDP1’s `no_sync()`.\n\nReference: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## Minimal reference implementation outline (agent-friendly)\n\nThe coding agent should implement a script with these labeled blocks:\n\n- `init_distributed()`: init process group, set device\n- `build_model_meta()`: model on meta, apply `fully_shard`, materialize weights\n- `build_optimizer()`: optimizer created after sharding\n- `train_step()`: forward\u002Fbackward\u002Fstep with `model(inputs)` and DTensor-aware patterns\n- `checkpoint_save\u002Fload()`: DCP or distributed state dict helpers\n\nConcrete examples live in `references\u002Fpytorch_examples_fsdp2.md` and the official tutorial reference.\n\n---\n\n## References\n- `references\u002Fpytorch_fsdp2_tutorial.md`\n- `references\u002Fpytorch_fully_shard_api.md`\n- `references\u002Fpytorch_ddp_notes.md`\n- `references\u002Fpytorch_fsdp1_api.md`\n- `references\u002Fpytorch_device_mesh_tutorial.md`\n- `references\u002Fpytorch_tp_tutorial.md`\n- `references\u002Fpytorch_dcp_overview.md`\n- `references\u002Fpytorch_dcp_recipe.md`\n- `references\u002Fpytorch_dcp_async_recipe.md`\n- `references\u002Fpytorch_examples_fsdp2.md`\n- `references\u002Ftorchtitan_fsdp_notes.md` (optional, production notes)\n- `references\u002Fray_train_fsdp2_example.md` (optional, integration example)\n","# Skill: Use PyTorch FSDP2 (`fully_shard`) correctly in a training script\n\nThis skill teaches a coding agent how to **add PyTorch FSDP2** to a training loop with correct initialization, sharding, mixed precision\u002Foffload configuration, and checkpointing.\n\n> FSDP2 in PyTorch is exposed primarily via `torch.distributed.fsdp.fully_shard` and the `FSDPModule` methods it adds in-place to modules. See: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## When to use this skill\n\nUse FSDP2 when:\n- Your model **doesn’t fit** on one GPU (parameters + gradients + optimizer state).\n- You want an eager-mode sharding approach that is **DTensor-based per-parameter sharding** (more inspectable, simpler sharded state dicts) than FSDP1.  \n- You may later compose DP with **Tensor Parallel** using **DeviceMesh**.\n\nAvoid (or be careful) if:\n- You need strict backwards-compatible checkpoints across PyTorch versions (DCP warns against this).\n- You’re forced onto older PyTorch versions without the FSDP2 stack.\n\n## Alternatives (when FSDP2 is not the best fit)\n\n- **DistributedDataParallel (DDP)**: Use the standard data-parallel wrapper when you want classic distributed data parallel training.\n- **FullyShardedDataParallel (FSDP1)**: Use the original FSDP wrapper for parameter sharding across data-parallel workers.\n\nReference: `references\u002Fpytorch_ddp_notes.md`, `references\u002Fpytorch_fsdp1_api.md`.\n\n---\n\n## Contract the agent must follow\n\n1. **Launch with `torchrun`** and set the CUDA device per process (usually via `LOCAL_RANK`).  \n2. **Apply `fully_shard()` bottom-up**, i.e., shard submodules (e.g., Transformer blocks) before the root module.  \n3. **Call `model(input)`**, not `model.forward(input)`, so the FSDP2 hooks run (unless you explicitly `unshard()` or register the forward method).  \n4. **Create the optimizer after sharding** and make sure it is built on the **DTensor parameters** (post-`fully_shard`).  \n5. **Checkpoint using Distributed Checkpoint (DCP)** or the distributed-state-dict helpers, not naïve `torch.save(model.state_dict())` unless you deliberately gather to full tensors.\n\n(Each of these rules is directly described in the official API docs\u002Ftutorial; see references.)\n\n---\n\n## Step-by-step procedure\n\n### 0) Version & environment sanity\n- Prefer a recent stable PyTorch where the docs show FSDP2 and DCP updated recently.\n- Use `torchrun --nproc_per_node \u003Cgpus_per_node> ...` and ensure `RANK`, `WORLD_SIZE`, `LOCAL_RANK` are visible.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (launch commands and setup), `references\u002Fpytorch_fully_shard_api.md` (user contract).\n\n---\n\n### 1) Initialize distributed and set device\nMinimal, correct pattern:\n- `dist.init_process_group(backend=\"nccl\")`\n- `torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))`\n- Optionally create a `DeviceMesh` to describe the data-parallel group(s)\n\nReference: `references\u002Fpytorch_device_mesh_tutorial.md` (why DeviceMesh exists & how it manages process groups).\n\n---\n\n### 2) Build model on meta device (recommended for very large models)\nFor big models, initialize on `meta`, apply sharding, then materialize weights on GPU:\n- `with torch.device(\"meta\"): model = ...`\n- apply `fully_shard(...)` on submodules, then `fully_shard(model)`\n- `model.to_empty(device=\"cuda\")`\n- `model.reset_parameters()` (or your init routine)\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (migration guide shows this flow explicitly).\n\n---\n\n### 3) Apply `fully_shard()` bottom-up (wrapping policy = “apply where needed”)\n**Do not** only call `fully_shard` on the topmost module.\n\nRecommended sharding pattern for transformer-like models:\n- iterate modules, `if isinstance(m, TransformerBlock): fully_shard(m, ...)`\n- then `fully_shard(model, ...)`\n\nWhy:\n- `fully_shard` forms “parameter groups” for collective efficiency and excludes params already grouped by earlier calls. Bottom-up gives better overlap and lower peak memory.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (bottom-up requirement and why).\n\n---\n\n### 4) Configure `reshard_after_forward` for memory\u002Fperf trade-offs\nDefault behavior:\n- `None` means `True` for non-root modules and `False` for root modules (good default).\n\nHeuristics:\n- If you’re memory-bound: keep defaults or force `True` on many blocks.\n- If you’re throughput-bound and can afford memory: consider keeping unsharded params longer (root often `False`).\n- Advanced: use an `int` to reshard to a smaller mesh after forward (e.g., intra-node) if it’s a meaningful divisor.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (full semantics).\n\n---\n\n### 5) Mixed precision & offload (optional but common)\nFSDP2 uses:\n- `mp_policy=MixedPrecisionPolicy(param_dtype=..., reduce_dtype=..., output_dtype=..., cast_forward_inputs=...)`\n- `offload_policy=CPUOffloadPolicy()` if you want CPU offload\n\nRules of thumb:\n- Start with BF16 parameters\u002Freductions on H100\u002FA100-class GPUs (if numerically stable for your model).\n- Keep `reduce_dtype` aligned with your gradient reduction expectations.\n- If you use CPU offload, budget for PCIe\u002FNVLink traffic and runtime overhead.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (MixedPrecisionPolicy \u002F OffloadPolicy classes).\n\n---\n\n### 6) Optimizer, gradient clipping, accumulation\n- Create the optimizer **after** sharding so it holds DTensor params.\n- If you need gradient accumulation \u002F no_sync:\n  - use the FSDP2 mechanism (`set_requires_gradient_sync`) instead of FSDP1’s `no_sync()`.\n\nGradient clipping:\n- Use the approach shown in the FSDP2 tutorial (“Gradient Clipping and Optimizer with DTensor”), because parameters\u002Fgradients are DTensors.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n### 7) Checkpointing: prefer DCP or distributed state dict helpers\nTwo recommended approaches:\n\n**A) Distributed Checkpoint (DCP) — best default**\n- DCP saves\u002Floads from multiple ranks in parallel and supports load-time resharding.\n- DCP produces **multiple files** (often at least one per rank) and operates “in place”.\n\n**B) Distributed state dict helpers**\n- `get_model_state_dict` \u002F `set_model_state_dict` with `StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True, ...)`\n- For optimizer: `get_optimizer_state_dict` \u002F `set_optimizer_state_dict`\n\nAvoid:\n- Saving DTensor state dicts with plain `torch.save` unless you intentionally convert with `DTensor.full_tensor()` and manage memory carefully.\n\nReferences:\n- `references\u002Fpytorch_dcp_overview.md` (DCP behavior and caveats)\n- `references\u002Fpytorch_dcp_recipe.md` and `references\u002Fpytorch_dcp_async_recipe.md` (end-to-end usage)\n- `references\u002Fpytorch_fsdp2_tutorial.md` (DTensor vs DCP state-dict flows)\n- `references\u002Fpytorch_examples_fsdp2.md` (working checkpoint scripts)\n\n---\n\n## Workflow checklists (copy-paste friendly)\n\n### Workflow A: Retrofit FSDP2 into an existing training script\n- [ ] Launch with `torchrun` and initialize the process group.\n- [ ] Set the CUDA device from `LOCAL_RANK`; create a `DeviceMesh` if you need multi-dim parallelism.\n- [ ] Build the model (use `meta` if needed), apply `fully_shard` bottom-up, then `fully_shard(model)`.\n- [ ] Create the optimizer after sharding so it captures DTensor parameters.\n- [ ] Use `model(inputs)` so hooks run; use `set_requires_gradient_sync` for accumulation.\n- [ ] Add DCP save\u002Fload via `torch.distributed.checkpoint` helpers.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`, `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_device_mesh_tutorial.md`, `references\u002Fpytorch_dcp_recipe.md`.\n\n### Workflow B: Add DCP save\u002Fload (minimal pattern)\n- [ ] Wrap state in `Stateful` or assemble state via `get_state_dict`.\n- [ ] Call `dcp.save(...)` from all ranks to a shared path.\n- [ ] Call `dcp.load(...)` and restore with `set_state_dict`.\n- [ ] Validate any resharding assumptions when loading into a different mesh.\n\nReference: `references\u002Fpytorch_dcp_recipe.md`.\n\n## Debug checklist (what the agent should check first)\n\n1. **All ranks on distinct GPUs?**  \n   If not, verify `torch.cuda.set_device(LOCAL_RANK)` and your `torchrun` flags.\n2. **Did you accidentally call `forward()` directly?**  \n   Use `model(input)` or explicitly `unshard()` \u002F register forward.\n3. **Is `fully_shard()` applied bottom-up?**  \n   If only root is sharded, expect worse memory\u002Fperf and possible confusion.\n4. **Optimizer created at the right time?**  \n   Must be built on DTensor parameters *after* sharding.\n5. **Checkpointing path consistent?**  \n   - If using DCP, don’t mix with ad-hoc `torch.save` unless you understand conversions.\n   - Be mindful of PyTorch-version compatibility warnings for DCP.\n\n---\n\n## Common issues and fixes\n\n- **Forward hooks not running** → Call `model(inputs)` (or `unshard()` explicitly) instead of `model.forward(...)`.\n- **Optimizer sees non-DTensor params** → Create optimizer after all `fully_shard` calls.\n- **Only root module sharded** → Apply `fully_shard` bottom-up on submodules before the root.\n- **Memory spikes after forward** → Set `reshard_after_forward=True` for more modules.\n- **Gradient accumulation desync** → Use `set_requires_gradient_sync` instead of FSDP1’s `no_sync()`.\n\nReference: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## Minimal reference implementation outline (agent-friendly)\n\nThe coding agent should implement a script with these labeled blocks:\n\n- `init_distributed()`: init process group, set device\n- `build_model_meta()`: model on meta, apply `fully_shard`, materialize weights\n- `build_optimizer()`: optimizer created after sharding\n- `train_step()`: forward\u002Fbackward\u002Fstep with `model(inputs)` and DTensor-aware patterns\n- `checkpoint_save\u002Fload()`: DCP or distributed state dict helpers\n\nConcrete examples live in `references\u002Fpytorch_examples_fsdp2.md` and the official tutorial reference.\n\n---\n\n## References\n- `references\u002Fpytorch_fsdp2_tutorial.md`\n- `references\u002Fpytorch_fully_shard_api.md`\n- `references\u002Fpytorch_ddp_notes.md`\n- `references\u002Fpytorch_fsdp1_api.md`\n- `references\u002Fpytorch_device_mesh_tutorial.md`\n- `references\u002Fpytorch_tp_tutorial.md`\n- `references\u002Fpytorch_dcp_overview.md`\n- `references\u002Fpytorch_dcp_recipe.md`\n- `references\u002Fpytorch_dcp_async_recipe.md`\n- `references\u002Fpytorch_examples_fsdp2.md`\n- `references\u002Ftorchtitan_fsdp_notes.md` (optional, production notes)\n- `references\u002Fray_train_fsdp2_example.md` (optional, integration example)\n",{"name":5,"tags":49,"author":54,"license":15,"version":55,"category":56,"description":7,"dependencies":57},[50,26,33,51,32,31,30,52,53,29],"PyTorch","Distributed Training","Mixed Precision","Offload","Orchestra Research","1.0.0","ai-research",[58],"torch","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Forch-pytorch-fsdp2","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":69,"prompt":79,"zipAvailable":80},[70,73,76],{"tool":71,"command":72},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill pytorch-fsdp2",{"tool":74,"command":75},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill pytorch-fsdp2",{"tool":77,"command":78},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill pytorch-fsdp2","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002FSKILL.md",true,[82,88,92,96,100,104,108,112,116,120,124,128],{"path":83,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fpytorch_dcp_async_recipe.md","REFERENCE","参考文档",808,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_async_recipe.md",{"path":89,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":90,"sourceHtmlUrl":91},"references\u002Fpytorch_dcp_overview.md",1022,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_overview.md",{"path":93,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Fpytorch_dcp_recipe.md",1043,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_recipe.md",{"path":97,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":98,"sourceHtmlUrl":99},"references\u002Fpytorch_ddp_notes.md",461,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_ddp_notes.md",{"path":101,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":102,"sourceHtmlUrl":103},"references\u002Fpytorch_device_mesh_tutorial.md",1222,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_device_mesh_tutorial.md",{"path":105,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":106,"sourceHtmlUrl":107},"references\u002Fpytorch_examples_fsdp2.md",742,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_examples_fsdp2.md",{"path":109,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":110,"sourceHtmlUrl":111},"references\u002Fpytorch_fsdp1_api.md",396,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_fsdp1_api.md",{"path":113,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":114,"sourceHtmlUrl":115},"references\u002Fpytorch_fsdp2_tutorial.md",2495,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_fsdp2_tutorial.md",{"path":117,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":118,"sourceHtmlUrl":119},"references\u002Fpytorch_fully_shard_api.md",2884,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_fully_shard_api.md",{"path":121,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":122,"sourceHtmlUrl":123},"references\u002Fpytorch_tp_tutorial.md",994,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fpytorch_tp_tutorial.md",{"path":125,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":126,"sourceHtmlUrl":127},"references\u002Fray_train_fsdp2_example.md",592,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Fray_train_fsdp2_example.md",{"path":129,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":130,"sourceHtmlUrl":131},"references\u002Ftorchtitan_fsdp_notes.md",666,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-pytorch-fsdp2\u002Freferences\u002Ftorchtitan_fsdp_notes.md",[133,163,186,211,235,259],{"skillNo":134,"name":135,"nameZh":6,"description":136,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":137,"categories":138,"keywords":142,"publishedAt":44},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[139],{"categoryNo":140,"categoryName":141,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000016","图像与多媒体",[143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","for","browsing","request","youtube transcripts",{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":169,"publishedAt":44},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[170,171,172,42,147,173,174,175,176,177,178,179,180,181,27,182,183,184,185,159],"twitter","x twitter growth","growth","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","platform","grow","posting","competitive",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":193,"publishedAt":44},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[192],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[194,180,195,196,197,198,199,200,27,201,202,59,203,204,205,206,207,208,209,210],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":212,"name":213,"nameZh":6,"description":214,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":215,"categories":216,"keywords":217,"publishedAt":44},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[218,219,220,221,222,223,224,225,226,27,227,154,228,229,230,231,28,232,233,234],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","tables","directions","phase",{"skillNo":236,"name":237,"nameZh":6,"description":238,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":239,"categories":240,"keywords":242,"publishedAt":44},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[241],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[243,244,245,246,247,196,223,248,249,250,251,252,180,253,153,254,255,256,257,258],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":260,"name":261,"nameZh":6,"description":262,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":263,"categories":264,"keywords":265,"publishedAt":44},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[266,267,268,269,270,271,272,273,274,251,275,276,277,278,279,280,155,281,282,234],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]