[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71NVWC0J5R849WSTA3TSS61":3,"skill-same:01M71NVWC0J5R849WSTA3TSS61":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":51,"install":52,"resources":65},"01M71NVWC0J5R849WSTA3TSS61","nemo-mbridge-perf-memory-tuning",null,"Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, PEFT + SP input re-gather, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.","pattanation\u002FNorvik","pattanation","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235419268?v=4","01M71M5THHYYB15ME8R2E4FCZG","Patta","01M71M5TJM1NDM992M8PDKYR2K",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"nemo mbridge perf memory tuning","tuning","perf","NeMo","memory","input","segments","sequence","config","CPU","change","common","commit","reduce","errors","pytorch","re","regression","bridge","megatron","2026-11-10 10:56:23",false,"---\nname: nemo-mbridge-perf-memory-tuning\ndescription: Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, PEFT + SP input re-gather, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.\nlicense: Apache-2.0\nwhen_to_use: GPU OOM errors, reducing peak memory, reducing LoRA or PEFT activation memory with sequence parallelism, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'LoRA memory', 'PEFT memory', 'sequence_parallel_input_regather', 'PYTORCH_CUDA_ALLOC_CONF'.\n---\n\n# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size, and PEFT-specific retention of gathered inputs\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **For LoRA with sequence parallelism, enable input re-gather**\n   (`LoRA(sequence_parallel_input_regather=True)`). This avoids retaining the\n   full gathered LoRA-A input in every eligible layer; it has no effect when SP\n   is disabled.\n3. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n4. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n5. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n6. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n7. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### PEFT + sequence-parallel input re-gather\n\nFor `LoRA` training with sequence parallelism, eligible column-parallel\n`linear_qkv` and `linear_fc1` adapters consume a gathered LayerNorm output.\nBecause LoRA-A is trainable, the default path retains that full gathered input\nuntil backward for the LoRA-A weight gradient.\n\nEnable input re-gather when constructing the PEFT config:\n\n```python\nfrom megatron.bridge.peft.lora import LoRA\n\ncfg.peft = LoRA(\n    # Keep the recipe's existing LoRA settings here.\n    sequence_parallel_input_regather=True,\n)\n```\n\nWith this option, forward still materializes the full input temporarily for the\nLoRA-A GEMM, but MCore autograd retains only its sequence-local shard. Backward\nasynchronously gathers the full input again, overlaps the collective with\ndgrad when possible, computes the LoRA-A weight gradient, and then reuses the\ntemporary communication buffer.\n\nThis is a memory-for-communication tradeoff, not conventional activation\ncheckpointing: no LayerNorm, attention, MLP, or LoRA GEMM is rerun. Some\nthroughput degradation is expected, and the benefit grows with the amount of\neligible LoRA-A activation retained. The option has no effect when sequence\nparallelism is disabled.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n- `sequence_parallel_input_regather` applies only to eligible non-expert\n  column-parallel LoRA-A projections. Row-parallel adapters, expert adapters,\n  TP=1, CUDA graphs, CPU activation offload, and overlapping full-layer or\n  selective MLP activation recompute fall back to the existing path.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n### LoRA + SP input re-gather\n\nReal-checkpoint H100 training with SQuAD showed lower peak memory in all tested\nconfigurations, with workload-dependent throughput cost:\n\n| Model\u002Fconfig | Baseline peak | Input re-gather peak | Memory saved | Throughput change |\n|---|---:|---:|---:|---:|\n| Qwen3-8B, TP2, seq 8192 | 47.545 GB | 42.814 GB | 4.731 GB (10.0%) | -6.74% |\n| Qwen3-30B-A3B, TP4\u002FEP4 | 29.890 GB | 28.321 GB | 1.569 GB (5.2%) | -2.89% |\n| GPT-OSS-120B, TP2\u002FEP8 | 52.185 GB | 51.371 GB | 0.814 GB (1.6%) | -0.34% |\n\nAll runs had finite losses with zero skipped or NaN iterations. Two-rank BF16\nand FP32 checks matched the baseline for outputs, input gradients, LoRA-A and\nLoRA-B gradients, and two-microbatch fused `main_grad` accumulation.\n\n## Code Anchors\n\n### LoRA sequence-parallel input re-gather\n\n```text\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Flora.py\n    LoRA.sequence_parallel_input_regather\n\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py\n    ParallelLinearAdapter._sequence_parallel_input_regather_eligibility()\n    ParallelLinearAdapter.forward()\n```\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| LoRA + SP retains unexpectedly high activation memory | full gathered LoRA-A inputs are retained until backward | check whether `cfg.peft.sequence_parallel_input_regather` is enabled and the target is eligible | set `LoRA(sequence_parallel_input_regather=True)`; verify fallback constraints |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n- LoRA input re-gather does not cover row-parallel or expert adapters and may\n  have negligible benefit when few eligible LoRA-A activations dominate memory\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n\nFor LoRA + SP input re-gather, run the focused configuration tests and the real\ntwo-rank MCore backward-parity test:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py -k \"sequence_parallel_input_regather\" \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora.py -k \"sequence_parallel_input_regather\"\n\nuv run python -m torch.distributed.run --nproc_per_node=2 -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora_sp_input_regather_distributed.py\n```\n","# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size, and PEFT-specific retention of gathered inputs\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **For LoRA with sequence parallelism, enable input re-gather**\n   (`LoRA(sequence_parallel_input_regather=True)`). This avoids retaining the\n   full gathered LoRA-A input in every eligible layer; it has no effect when SP\n   is disabled.\n3. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n4. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n5. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n6. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n7. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### PEFT + sequence-parallel input re-gather\n\nFor `LoRA` training with sequence parallelism, eligible column-parallel\n`linear_qkv` and `linear_fc1` adapters consume a gathered LayerNorm output.\nBecause LoRA-A is trainable, the default path retains that full gathered input\nuntil backward for the LoRA-A weight gradient.\n\nEnable input re-gather when constructing the PEFT config:\n\n```python\nfrom megatron.bridge.peft.lora import LoRA\n\ncfg.peft = LoRA(\n    # Keep the recipe's existing LoRA settings here.\n    sequence_parallel_input_regather=True,\n)\n```\n\nWith this option, forward still materializes the full input temporarily for the\nLoRA-A GEMM, but MCore autograd retains only its sequence-local shard. Backward\nasynchronously gathers the full input again, overlaps the collective with\ndgrad when possible, computes the LoRA-A weight gradient, and then reuses the\ntemporary communication buffer.\n\nThis is a memory-for-communication tradeoff, not conventional activation\ncheckpointing: no LayerNorm, attention, MLP, or LoRA GEMM is rerun. Some\nthroughput degradation is expected, and the benefit grows with the amount of\neligible LoRA-A activation retained. The option has no effect when sequence\nparallelism is disabled.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n- `sequence_parallel_input_regather` applies only to eligible non-expert\n  column-parallel LoRA-A projections. Row-parallel adapters, expert adapters,\n  TP=1, CUDA graphs, CPU activation offload, and overlapping full-layer or\n  selective MLP activation recompute fall back to the existing path.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n### LoRA + SP input re-gather\n\nReal-checkpoint H100 training with SQuAD showed lower peak memory in all tested\nconfigurations, with workload-dependent throughput cost:\n\n| Model\u002Fconfig | Baseline peak | Input re-gather peak | Memory saved | Throughput change |\n|---|---:|---:|---:|---:|\n| Qwen3-8B, TP2, seq 8192 | 47.545 GB | 42.814 GB | 4.731 GB (10.0%) | -6.74% |\n| Qwen3-30B-A3B, TP4\u002FEP4 | 29.890 GB | 28.321 GB | 1.569 GB (5.2%) | -2.89% |\n| GPT-OSS-120B, TP2\u002FEP8 | 52.185 GB | 51.371 GB | 0.814 GB (1.6%) | -0.34% |\n\nAll runs had finite losses with zero skipped or NaN iterations. Two-rank BF16\nand FP32 checks matched the baseline for outputs, input gradients, LoRA-A and\nLoRA-B gradients, and two-microbatch fused `main_grad` accumulation.\n\n## Code Anchors\n\n### LoRA sequence-parallel input re-gather\n\n```text\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Flora.py\n    LoRA.sequence_parallel_input_regather\n\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py\n    ParallelLinearAdapter._sequence_parallel_input_regather_eligibility()\n    ParallelLinearAdapter.forward()\n```\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| LoRA + SP retains unexpectedly high activation memory | full gathered LoRA-A inputs are retained until backward | check whether `cfg.peft.sequence_parallel_input_regather` is enabled and the target is eligible | set `LoRA(sequence_parallel_input_regather=True)`; verify fallback constraints |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n- LoRA input re-gather does not cover row-parallel or expert adapters and may\n  have negligible benefit when few eligible LoRA-A activations dominate memory\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n\nFor LoRA + SP input re-gather, run the focused configuration tests and the real\ntwo-rank MCore backward-parity test:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py -k \"sequence_parallel_input_regather\" \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora.py -k \"sequence_parallel_input_regather\"\n\nuv run python -m torch.distributed.run --nproc_per_node=2 -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora_sp_input_regather_distributed.py\n```\n",{"name":5,"license":15,"description":7,"when_to_use":44},"GPU OOM errors, reducing peak memory, reducing LoRA or PEFT activation memory with sequence parallelism, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'LoRA memory', 'PEFT memory', 'sequence_parallel_input_regather', 'PYTORCH_CUDA_ALLOC_CONF'.","claude\u002Finstall-gsd-build-foQdV","d86cd127f527124b5fd0fd0bc679a4d79b7d01d7",".agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik","https:\u002F\u002Fnorvik-orpin.vercel.app","https:\u002F\u002Fgithub.com\u002Fpattanation","2026-07-17 11:17:04",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add pattanation\u002FNorvik --skill nemo-mbridge-perf-memory-tuning",{"tool":58,"command":59},"bunx","bunx skills add pattanation\u002FNorvik --skill nemo-mbridge-perf-memory-tuning",{"tool":61,"command":62},"pnpm","pnpm dlx skills add pattanation\u002FNorvik --skill nemo-mbridge-perf-memory-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FSKILL.md",true,[66,72,76,80,84],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"BENCHMARK.md","REFERENCE","参考文档",3997,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FBENCHMARK.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"card.yaml",10029,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"evals\u002Fevals.json",1474,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fevals\u002Fevals.json",{"path":81,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":82,"sourceHtmlUrl":83},"skill-card.md",4162,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fskill-card.md",{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"skill.oms.sig","OTHER","其他",4825,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fskill.oms.sig",[91,125,151,175,197,212],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":106,"publishedAt":124},"01M71QDV4DM8T19HEHWATEF1K4","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[98,103],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000009","前端开发",[107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123],"webapp","testing","webapp testing","for","applications","local","frontend","debugging","toolkit","logs","browser","with","playwright","web","behavior","ui","screenshots","2026-11-10 13:02:02",{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":129,"sourceHtmlUrl":130,"categories":131,"keywords":133,"publishedAt":124},"01M71QDH2GGMXGA9105C9W1N7B","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[132],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[134,135,136,120,137,138,139,110,140,141,113,142,143,144,145,146,147,148,149,150],"builder","artifacts builder","artifacts","web artifacts builder","single-file","css","Tailwind","ui components","single","use","file","components","react","multi","state","routing","state management",{"skillNo":152,"name":153,"nameZh":6,"description":154,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":155,"licenseFileUrl":156,"sourceHtmlUrl":157,"categories":158,"keywords":160,"publishedAt":124},"01M71QD9T2XC3EWMQNN9ACNWQ1","vueuse-functions","Apply VueUse composables where appropriate to build concise, maintainable Vue.js \u002F Nuxt features.","MIT","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvueuse-functions\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvueuse-functions\u002FSKILL.md",[159],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[161,162,163,164,165,166,167,168,169,170,171,172,173,174],"functions","vueuse","vueuse functions","project","nuxt","apply","github","concise","build","https","features","vue","js","composables",{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":155,"licenseFileUrl":179,"sourceHtmlUrl":180,"categories":181,"keywords":184,"publishedAt":124},"01M71QD51HWSGYMCMB7RW7PGK8","vue-testing-best-practices","Use for Vue.js testing. Covers Vitest, Vue Test Utils, component testing, mocking, testing patterns, and Playwright for E2E testing.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-testing-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-testing-best-practices\u002FSKILL.md",[182,183],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[185,108,186,172,187,119,188,189,190,167,191,110,192,193,194,143,195,173,196],"vue testing best practices","testing best practices","best practices","test","testing patterns","component","vitest","mocking","e2e","utils","patterns","e2e testing",{"skillNo":198,"name":199,"nameZh":6,"description":200,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":155,"licenseFileUrl":201,"sourceHtmlUrl":202,"categories":203,"keywords":205,"publishedAt":124},"01M71QD358H2DJ9THJRBP77T3T","vue-router-best-practices","Vue Router 4 patterns, navigation guards, route params, and route-component lifecycle interactions.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-router-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-router-best-practices\u002FSKILL.md",[204],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[206,172,207,187,208,190,209,167,195,210,211],"router","vue router best practices","route","guards","navigation","lifecycle",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":155,"licenseFileUrl":216,"sourceHtmlUrl":217,"categories":218,"keywords":220,"publishedAt":124},"01M71QCYPNWDE2MQZ1TF3ED5S4","vue-best-practices","MUST be used for Vue.js tasks. Strongly recommends Composition API with `\u003Cscript setup>` and TypeScript as the standard approach. Covers Vue 3, SSR, Volar, vue-tsc. Load for any Vue, .vue files, Vue Router, Pinia, or Vite with Vue work. ALWAYS use Composition API unless the project explicitly requires Options API.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-best-practices\u002FSKILL.md",[219],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[221,172,187,222,164,223,224,225,206,226,227,228,229,230,231,110,232,143,233,234],"vue best practices","Composition","files","tasks","api","pinia","work","options","ssr","typescript","as","vite","load","standard"]