[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CSMD9R6D6JWSBA1NBTH83Q":3,"skill-same:01M7CSMD9R6D6JWSBA1NBTH83Q":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7CSMD9R6D6JWSBA1NBTH83Q","nemo-mbridge-perf-memory-tuning",null,"Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-memory-tuning\ndescription: Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.\nlicense: Apache-2.0\nwhen_to_use: GPU OOM errors, reducing peak memory, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'PYTORCH_CUDA_ALLOC_CONF'.\n---\n\n# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n3. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n4. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n5. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n6. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n## Code Anchors\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n","# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n3. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n4. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n5. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n6. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n## Code Anchors\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n",{"name":5,"license":15,"description":7,"when_to_use":24},"GPU OOM errors, reducing peak memory, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'PYTORCH_CUDA_ALLOC_CONF'.","main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill nemo-mbridge-perf-memory-tuning",{"tool":40,"command":41},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill nemo-mbridge-perf-memory-tuning",{"tool":43,"command":44},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill nemo-mbridge-perf-memory-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FSKILL.md",true,[48,54,58,62,66],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"BENCHMARK.md","REFERENCE","参考文档",4331,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FBENCHMARK.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"card.yaml",6902,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"evals\u002Fevals.json",1474,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fevals\u002Fevals.json",{"path":63,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":64,"sourceHtmlUrl":65},"skill-card.md",3882,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fskill-card.md",{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"skill.oms.sig","OTHER","其他",4825,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fskill.oms.sig",[73,81,88,95,102,109],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]