[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CPVK49NZHWY3C1G34TYRVR":3,"skill-same:01M7CPVK49NZHWY3C1G34TYRVR":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7CPVK49NZHWY3C1G34TYRVR","nemo-mbridge-perf-expert-parallel-overlap",null,"Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including overlap_moe_expert_parallel_comm, delay_wgrad_compute, and flex dispatcher backends such as DeepEP and HybridEP.","om-scogo\u002Fskillsh-scraper","om-scogo","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160466840?v=4","01M79AWB6DFVCQ7RK70CS34B96","Om Achrekar","01M79AWB6ZM9SKR8KDH1WSSM22",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fnvidia-skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fskills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-expert-parallel-overlap\ndescription: Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including overlap_moe_expert_parallel_comm, delay_wgrad_compute, and flex dispatcher backends such as DeepEP and HybridEP.\nlicense: Apache-2.0\nwhen_to_use: Enabling EP overlap to hide dispatch\u002Fcombine latency, or tracing a throughput regression to an EP overlap config change; 'overlap_moe_expert_parallel_comm', 'delay_wgrad_compute', 'flex dispatcher', 'DeepEP overlap', 'HybridEP overlap'.\n---\n\n# MoE Expert-Parallel Overlap Skill\n\n## References\n\n- Stable docs: @docs\u002Ftraining\u002Fcommunication-overlap.md\n- Structured metadata: @skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fcard.yaml\n\n## What It Is\n\nExpert-parallel (EP) overlap hides the cost of token dispatch\u002Fcombine all-to-all\ncommunication by running it concurrently with expert FFN compute. Optionally,\ndelayed expert weight-gradient computation (`delay_wgrad_compute`) provides\nadditional overlap by deferring wgrad to overlap with the next layer's forward.\n\nBridge supports two dispatcher paths:\n\n| Dispatcher | Backend | When to use |\n|---|---|---|\n| `alltoall` | Standard MoE all-to-all | Default, broadest compatibility |\n| `flex` | DeepEP or HybridEP | Higher overlap on Ampere\u002FHopper\u002FBlackwell |\n\n## Quick Decision\n\nUse EP overlap when:\n\n- the model is MoE with `EP > 1`\n- expert dispatch\u002Fcombine communication is a meaningful part of step time\n- you have memory headroom and are tuning for throughput\n\nPrefer:\n\n- `alltoall` dispatcher for the first rollout (broader compatibility)\n- `flex` + DeepEP\u002FHybridEP when running on supported GPUs and seeking\n  additional gains\n\nAvoid EP overlap when:\n\n- full activation recompute is enabled\n- `moe_shared_expert_overlap` is enabled\n- the run is still being brought up for correctness\n- PyTorch \u003C 2.6.0\n\nExpected outcome:\n\n- if all-to-all dispatch is a clear profile bottleneck, overlap can produce a\n  modest to meaningful speedup\n- if the run is tiny, communication-light, or dominated by another wall, the\n  gain may be negligible\n\n## Correctness-First alltoall Benchmark\n\nFor the plain EP-overlap isolation benchmark, keep flex dispatch and delayed\nwgrad disabled. The measured shape was Qwen3 MoE 30B-A3B SFT on 16 H100 GPUs:\n`EP=16`, `alltoall`, BF16, global batch size 1024, CUDA graphs disabled,\n`moe_permute_fusion=false`, measured over iterations 3-8.\n\nUse these overrides for the plain-overlap case:\n\n```bash\n--cuda_graph_impl none \\\n--moe_flex_dispatcher_backend None \\\n--moe_a2a_overlap false \\\ncomm_overlap.overlap_moe_expert_parallel_comm=true \\\ncomm_overlap.delay_wgrad_compute=false \\\nmodel.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` for this isolation test: the performance\nharness helper enables both `overlap_moe_expert_parallel_comm` and\n`delay_wgrad_compute`, so it does not isolate plain EP overlap.\n\nSteady-window timing from that benchmark:\n\n| Case | Steady mean | Relative |\n|---|---:|---:|\n| no EP overlap | 41.25s | 1.000x |\n| EP overlap | 31.31s | 1.317x |\n| EP overlap plus `delay_wgrad_compute` | 31.20s | 1.322x |\n\nThis is evidence for enabling plain EP overlap on this inter-node all-to-all\nshape. It does not show a meaningful independent win from delayed wgrad, and it\ndoes not validate fused MoE permutation because that path was disabled for the\nruntime stack.\n\n## Enablement\n\n### alltoall dispatcher\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.moe_shared_expert_overlap = False\n\ncfg.model.expert_model_parallel_size = 8\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.bf16 = True\ncfg.model.fp16 = False\n```\n\nEnable `delay_wgrad_compute=True` only after the plain overlap path is known to\nwork and its extra compatibility constraints have been checked.\n\n### flex dispatcher (DeepEP or HybridEP)\n\n```python\nfrom megatron.bridge.training.flex_dispatcher_backend import apply_flex_dispatcher_backend\n\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = True\ncfg.model.moe_shared_expert_overlap = False\n\napply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"deepep\")\n# or: apply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"hybridep\")\n```\n\n## Compatibility And Constraints\n\n- `expert_model_parallel_size > 1`\n- `num_moe_experts > 1`\n- `moe_token_dispatcher_type` must be `\"alltoall\"` or `\"flex\"`\n- `moe_shared_expert_overlap = False`\n- Base precision is BF16 or FP16\n- PyTorch `>= 2.6.0`\n- If `PP > 1`, `virtual_pipeline_model_parallel_size` must be set\n- `recompute_granularity != \"full\"`, `recompute_method = None`,\n  `recompute_num_layers = None`\n- `mtp_num_layers` must be `None` or `1`\n- `delay_wgrad_compute` requires `overlap_moe_expert_parallel_comm` as a\n  prerequisite\n- `delay_wgrad_compute` with `overlap_grad_reduce` requires TE >= 2.7.0\n- `delay_wgrad_compute` with `gradient_accumulation_fusion` requires TE >= 2.7.0\n- CUDA graph `attn` scope + `delay_wgrad_compute` requires TE >= 2.12.0,\n  `gradient_accumulation_fusion = True`, and no attention bias\n- DeepEP: Ampere, Hopper, B200, B300 GPUs only\n- HybridEP: Ampere, Hopper, B200, B300, GB200\u002FGB300 with NVL72\n\n## Minimal Working Config\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.expert_model_parallel_size = 4\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.moe_shared_expert_overlap = False\ncfg.model.bf16 = True\n```\n\nUse this as the correctness-first starting point. Add delayed wgrad, flex\ndispatch, and CUDA-graph interactions only after the plain overlap path is\nknown to work.\n\n## Minimal Runnable Command\n\nPerformance harness example inside a Slurm allocation. Keep the model,\nparallelism, dispatcher, and runtime fixed, and vary only the two overlap\noverrides:\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  -gn 8 \\\n  --max_steps 8 \\\n  --cuda_graph_impl none \\\n  --moe_flex_dispatcher_backend None \\\n  --moe_a2a_overlap false \\\n  --tokenizer_type NullTokenizer \\\n  comm_overlap.overlap_moe_expert_parallel_comm=true \\\n  comm_overlap.delay_wgrad_compute=false \\\n  model.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` when separating plain EP overlap from\ndelayed wgrad: the performance harness helper enables both\n`overlap_moe_expert_parallel_comm` and `delay_wgrad_compute`.\n\nUnit test verification:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py -k \"moe\" \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n## Verification\n\n### Unit tests\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n### Log checks\n\nAfter a successful run with EP overlap:\n\n1. Confirm no assertion errors during `CommOverlapConfig` finalization\n2. Confirm `overlap_moe_expert_parallel_comm` appears as `True` in the logged\n   config\n3. If using flex dispatcher, confirm `moe_token_dispatcher_type = \"flex\"` and\n   the correct backend in logs\n\n### Success criteria\n\n- Config validation passes for the selected dispatcher and overlap settings\n- Training runs complete without hangs or assertion failures\n- Throughput improves or at least does not regress for the target workload\n- Loss trajectory matches baseline (overlap should not affect convergence)\n\n## Code Anchors\n\n### Bridge overlap validation\n\n```470:505:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.overlap_moe_expert_parallel_comm is True:\n    assert model_cfg.expert_model_parallel_size > 1, ...\n    assert model_cfg.num_moe_experts > 1, ...\n    assert model_cfg.moe_token_dispatcher_type in [\"alltoall\", \"flex\"], ...\n    assert model_cfg.bf16 or model_cfg.fp16, ...\n    assert is_torch_min_version(\"2.6.0\"), ...\n    # ... PP + VPP check, recompute checks, shared_expert_overlap check ...\n```\n\n### Delayed wgrad validation\n\n```507:557:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.delay_wgrad_compute is True:\n    # TE version checks for overlap_grad_reduce and gradient_accumulation_fusion\n    # CUDA graph scope validations for delayed wgrad\n    assert overlap_moe_expert_parallel_comm, ...\n```\n\n### Flex-dispatcher activation\n\n```27:72:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fflex_dispatcher_backend.py\ndef apply_flex_dispatcher_backend(...):\n    # GPU architecture check for DeepEP \u002F HybridEP\n    model_config.moe_token_dispatcher_type = \"flex\"\n    model_config.moe_flex_dispatcher_backend = moe_flex_dispatcher_backend\n    model_config.moe_shared_expert_overlap = False\n```\n\n### Perf harness override\n\n```149:156:scripts\u002Fperformance\u002Futils\u002Foverrides.py\ndef _set_moe_a2a_overlap_overrides(recipe, moe_a2a_overlap=False):\n    if moe_a2a_overlap:\n        recipe.comm_overlap.overlap_moe_expert_parallel_comm = True\n        recipe.comm_overlap.delay_wgrad_compute = True\n        recipe.model.moe_shared_expert_overlap = False\n```\n\n### Tests\n\n| File | Coverage |\n|---|---|\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py` | EP overlap validation, delayed wgrad, CUDA graph + wgrad interaction |\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py` | DeepEP\u002FHybridEP helper activation and GPU gating |\n\n## Failure Diagnosis\n\n| Symptom | Likely Cause | How To Confirm | Fix |\n|---|---|---|---|\n| assert `expert_model_parallel_size > 1` | EP not configured | Check `expert_model_parallel_size` | Set EP > 1 |\n| assert `moe_token_dispatcher_type` | Wrong dispatcher | Check dispatcher type | Use `\"alltoall\"` or `\"flex\"` |\n| assert on BF16\u002FFP16 | Wrong precision | Check `bf16` and `fp16` | Set `bf16 = True` |\n| hang during training | PyTorch \u003C 2.6 | Check PyTorch version | Upgrade to >= 2.6.0 |\n| assert `virtual_pipeline_model_parallel_size` | PP > 1 without VPP | Check PP and VPP config | Set VPP when PP > 1 |\n| assert `recompute_granularity` | Full recompute enabled | Check recompute settings | Disable full recompute |\n| assert `overlap_moe_expert_parallel_comm required` | delayed wgrad without EP overlap | Check `delay_wgrad_compute` without overlap | Enable EP overlap first |\n| assert `gradient_accumulation_fusion` | CUDA graph + delayed wgrad | Check graph scope + wgrad settings | Enable `gradient_accumulation_fusion` |\n| assert on attention bias | CUDA graph attn + delayed wgrad + bias | Check `add_bias_linear` \u002F `add_qkv_bias` | Disable attention bias |\n| no throughput gain from flex dispatcher | `apply_flex_dispatcher_backend` not called | Check `moe_token_dispatcher_type` in logs | Call `apply_flex_dispatcher_backend(...)` |\n| DeepEP\u002FHybridEP silently skipped | Unsupported GPU | Check warning logs | Run on Ampere\u002FHopper\u002FBlackwell |\n\n## Known Limitations\n\n- Setting `moe_flex_dispatcher_backend` alone does not activate flex dispatch —\n  you must call `apply_flex_dispatcher_backend(...)`.\n- Public recipes are often conservative and leave MoE overlap disabled by\n  default.\n- End-to-end throughput gains have not yet been measured in a controlled Bridge\n  experiment for every model family. Code validation is stronger than a single\n  universal performance claim.\n- MoE overlap and shared-expert overlap are mutually exclusive.\n- CUDA graph plus delayed wgrad is a multi-constraint path that requires\n  careful TE version and scope validation.\n","# MoE Expert-Parallel Overlap Skill\n\n## References\n\n- Stable docs: @docs\u002Ftraining\u002Fcommunication-overlap.md\n- Structured metadata: @skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fcard.yaml\n\n## What It Is\n\nExpert-parallel (EP) overlap hides the cost of token dispatch\u002Fcombine all-to-all\ncommunication by running it concurrently with expert FFN compute. Optionally,\ndelayed expert weight-gradient computation (`delay_wgrad_compute`) provides\nadditional overlap by deferring wgrad to overlap with the next layer's forward.\n\nBridge supports two dispatcher paths:\n\n| Dispatcher | Backend | When to use |\n|---|---|---|\n| `alltoall` | Standard MoE all-to-all | Default, broadest compatibility |\n| `flex` | DeepEP or HybridEP | Higher overlap on Ampere\u002FHopper\u002FBlackwell |\n\n## Quick Decision\n\nUse EP overlap when:\n\n- the model is MoE with `EP > 1`\n- expert dispatch\u002Fcombine communication is a meaningful part of step time\n- you have memory headroom and are tuning for throughput\n\nPrefer:\n\n- `alltoall` dispatcher for the first rollout (broader compatibility)\n- `flex` + DeepEP\u002FHybridEP when running on supported GPUs and seeking\n  additional gains\n\nAvoid EP overlap when:\n\n- full activation recompute is enabled\n- `moe_shared_expert_overlap` is enabled\n- the run is still being brought up for correctness\n- PyTorch \u003C 2.6.0\n\nExpected outcome:\n\n- if all-to-all dispatch is a clear profile bottleneck, overlap can produce a\n  modest to meaningful speedup\n- if the run is tiny, communication-light, or dominated by another wall, the\n  gain may be negligible\n\n## Correctness-First alltoall Benchmark\n\nFor the plain EP-overlap isolation benchmark, keep flex dispatch and delayed\nwgrad disabled. The measured shape was Qwen3 MoE 30B-A3B SFT on 16 H100 GPUs:\n`EP=16`, `alltoall`, BF16, global batch size 1024, CUDA graphs disabled,\n`moe_permute_fusion=false`, measured over iterations 3-8.\n\nUse these overrides for the plain-overlap case:\n\n```bash\n--cuda_graph_impl none \\\n--moe_flex_dispatcher_backend None \\\n--moe_a2a_overlap false \\\ncomm_overlap.overlap_moe_expert_parallel_comm=true \\\ncomm_overlap.delay_wgrad_compute=false \\\nmodel.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` for this isolation test: the performance\nharness helper enables both `overlap_moe_expert_parallel_comm` and\n`delay_wgrad_compute`, so it does not isolate plain EP overlap.\n\nSteady-window timing from that benchmark:\n\n| Case | Steady mean | Relative |\n|---|---:|---:|\n| no EP overlap | 41.25s | 1.000x |\n| EP overlap | 31.31s | 1.317x |\n| EP overlap plus `delay_wgrad_compute` | 31.20s | 1.322x |\n\nThis is evidence for enabling plain EP overlap on this inter-node all-to-all\nshape. It does not show a meaningful independent win from delayed wgrad, and it\ndoes not validate fused MoE permutation because that path was disabled for the\nruntime stack.\n\n## Enablement\n\n### alltoall dispatcher\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.moe_shared_expert_overlap = False\n\ncfg.model.expert_model_parallel_size = 8\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.bf16 = True\ncfg.model.fp16 = False\n```\n\nEnable `delay_wgrad_compute=True` only after the plain overlap path is known to\nwork and its extra compatibility constraints have been checked.\n\n### flex dispatcher (DeepEP or HybridEP)\n\n```python\nfrom megatron.bridge.training.flex_dispatcher_backend import apply_flex_dispatcher_backend\n\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = True\ncfg.model.moe_shared_expert_overlap = False\n\napply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"deepep\")\n# or: apply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"hybridep\")\n```\n\n## Compatibility And Constraints\n\n- `expert_model_parallel_size > 1`\n- `num_moe_experts > 1`\n- `moe_token_dispatcher_type` must be `\"alltoall\"` or `\"flex\"`\n- `moe_shared_expert_overlap = False`\n- Base precision is BF16 or FP16\n- PyTorch `>= 2.6.0`\n- If `PP > 1`, `virtual_pipeline_model_parallel_size` must be set\n- `recompute_granularity != \"full\"`, `recompute_method = None`,\n  `recompute_num_layers = None`\n- `mtp_num_layers` must be `None` or `1`\n- `delay_wgrad_compute` requires `overlap_moe_expert_parallel_comm` as a\n  prerequisite\n- `delay_wgrad_compute` with `overlap_grad_reduce` requires TE >= 2.7.0\n- `delay_wgrad_compute` with `gradient_accumulation_fusion` requires TE >= 2.7.0\n- CUDA graph `attn` scope + `delay_wgrad_compute` requires TE >= 2.12.0,\n  `gradient_accumulation_fusion = True`, and no attention bias\n- DeepEP: Ampere, Hopper, B200, B300 GPUs only\n- HybridEP: Ampere, Hopper, B200, B300, GB200\u002FGB300 with NVL72\n\n## Minimal Working Config\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.expert_model_parallel_size = 4\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.moe_shared_expert_overlap = False\ncfg.model.bf16 = True\n```\n\nUse this as the correctness-first starting point. Add delayed wgrad, flex\ndispatch, and CUDA-graph interactions only after the plain overlap path is\nknown to work.\n\n## Minimal Runnable Command\n\nPerformance harness example inside a Slurm allocation. Keep the model,\nparallelism, dispatcher, and runtime fixed, and vary only the two overlap\noverrides:\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  -gn 8 \\\n  --max_steps 8 \\\n  --cuda_graph_impl none \\\n  --moe_flex_dispatcher_backend None \\\n  --moe_a2a_overlap false \\\n  --tokenizer_type NullTokenizer \\\n  comm_overlap.overlap_moe_expert_parallel_comm=true \\\n  comm_overlap.delay_wgrad_compute=false \\\n  model.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` when separating plain EP overlap from\ndelayed wgrad: the performance harness helper enables both\n`overlap_moe_expert_parallel_comm` and `delay_wgrad_compute`.\n\nUnit test verification:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py -k \"moe\" \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n## Verification\n\n### Unit tests\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n### Log checks\n\nAfter a successful run with EP overlap:\n\n1. Confirm no assertion errors during `CommOverlapConfig` finalization\n2. Confirm `overlap_moe_expert_parallel_comm` appears as `True` in the logged\n   config\n3. If using flex dispatcher, confirm `moe_token_dispatcher_type = \"flex\"` and\n   the correct backend in logs\n\n### Success criteria\n\n- Config validation passes for the selected dispatcher and overlap settings\n- Training runs complete without hangs or assertion failures\n- Throughput improves or at least does not regress for the target workload\n- Loss trajectory matches baseline (overlap should not affect convergence)\n\n## Code Anchors\n\n### Bridge overlap validation\n\n```470:505:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.overlap_moe_expert_parallel_comm is True:\n    assert model_cfg.expert_model_parallel_size > 1, ...\n    assert model_cfg.num_moe_experts > 1, ...\n    assert model_cfg.moe_token_dispatcher_type in [\"alltoall\", \"flex\"], ...\n    assert model_cfg.bf16 or model_cfg.fp16, ...\n    assert is_torch_min_version(\"2.6.0\"), ...\n    # ... PP + VPP check, recompute checks, shared_expert_overlap check ...\n```\n\n### Delayed wgrad validation\n\n```507:557:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.delay_wgrad_compute is True:\n    # TE version checks for overlap_grad_reduce and gradient_accumulation_fusion\n    # CUDA graph scope validations for delayed wgrad\n    assert overlap_moe_expert_parallel_comm, ...\n```\n\n### Flex-dispatcher activation\n\n```27:72:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fflex_dispatcher_backend.py\ndef apply_flex_dispatcher_backend(...):\n    # GPU architecture check for DeepEP \u002F HybridEP\n    model_config.moe_token_dispatcher_type = \"flex\"\n    model_config.moe_flex_dispatcher_backend = moe_flex_dispatcher_backend\n    model_config.moe_shared_expert_overlap = False\n```\n\n### Perf harness override\n\n```149:156:scripts\u002Fperformance\u002Futils\u002Foverrides.py\ndef _set_moe_a2a_overlap_overrides(recipe, moe_a2a_overlap=False):\n    if moe_a2a_overlap:\n        recipe.comm_overlap.overlap_moe_expert_parallel_comm = True\n        recipe.comm_overlap.delay_wgrad_compute = True\n        recipe.model.moe_shared_expert_overlap = False\n```\n\n### Tests\n\n| File | Coverage |\n|---|---|\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py` | EP overlap validation, delayed wgrad, CUDA graph + wgrad interaction |\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py` | DeepEP\u002FHybridEP helper activation and GPU gating |\n\n## Failure Diagnosis\n\n| Symptom | Likely Cause | How To Confirm | Fix |\n|---|---|---|---|\n| assert `expert_model_parallel_size > 1` | EP not configured | Check `expert_model_parallel_size` | Set EP > 1 |\n| assert `moe_token_dispatcher_type` | Wrong dispatcher | Check dispatcher type | Use `\"alltoall\"` or `\"flex\"` |\n| assert on BF16\u002FFP16 | Wrong precision | Check `bf16` and `fp16` | Set `bf16 = True` |\n| hang during training | PyTorch \u003C 2.6 | Check PyTorch version | Upgrade to >= 2.6.0 |\n| assert `virtual_pipeline_model_parallel_size` | PP > 1 without VPP | Check PP and VPP config | Set VPP when PP > 1 |\n| assert `recompute_granularity` | Full recompute enabled | Check recompute settings | Disable full recompute |\n| assert `overlap_moe_expert_parallel_comm required` | delayed wgrad without EP overlap | Check `delay_wgrad_compute` without overlap | Enable EP overlap first |\n| assert `gradient_accumulation_fusion` | CUDA graph + delayed wgrad | Check graph scope + wgrad settings | Enable `gradient_accumulation_fusion` |\n| assert on attention bias | CUDA graph attn + delayed wgrad + bias | Check `add_bias_linear` \u002F `add_qkv_bias` | Disable attention bias |\n| no throughput gain from flex dispatcher | `apply_flex_dispatcher_backend` not called | Check `moe_token_dispatcher_type` in logs | Call `apply_flex_dispatcher_backend(...)` |\n| DeepEP\u002FHybridEP silently skipped | Unsupported GPU | Check warning logs | Run on Ampere\u002FHopper\u002FBlackwell |\n\n## Known Limitations\n\n- Setting `moe_flex_dispatcher_backend` alone does not activate flex dispatch —\n  you must call `apply_flex_dispatcher_backend(...)`.\n- Public recipes are often conservative and leave MoE overlap disabled by\n  default.\n- End-to-end throughput gains have not yet been measured in a controlled Bridge\n  experiment for every model family. Code validation is stronger than a single\n  universal performance claim.\n- MoE overlap and shared-expert overlap are mutually exclusive.\n- CUDA graph plus delayed wgrad is a multi-constraint path that requires\n  careful TE version and scope validation.\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Enabling EP overlap to hide dispatch\u002Fcombine latency, or tracing a throughput regression to an EP overlap config change; 'overlap_moe_expert_parallel_comm', 'delay_wgrad_compute', 'flex dispatcher', 'DeepEP overlap', 'HybridEP overlap'.","main","8859218955d9504cd18b59e7892100718039d3b4","data\u002Fskills\u002Fnvidia-skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fskills\u002Fnemo-mbridge-perf-expert-parallel-overlap",1,"https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper","https:\u002F\u002Fgithub.com\u002Fom-scogo","SDE-1 @ Scogo ","2026-07-18 12:22:46",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add om-scogo\u002Fskillsh-scraper --skill nemo-mbridge-perf-expert-parallel-overlap",{"tool":39,"command":40},"bunx","bunx skills add om-scogo\u002Fskillsh-scraper --skill nemo-mbridge-perf-expert-parallel-overlap",{"tool":42,"command":43},"pnpm","pnpm dlx skills add om-scogo\u002Fskillsh-scraper --skill nemo-mbridge-perf-expert-parallel-overlap","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fnvidia-skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fskills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002FSKILL.md",true,[],[48,56,61,68,73,80],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":19},"01M7D6MC9FAAEV4CMNYX8W7JTG","arxiv-search","Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.","MIT","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fyorkeccak-scientific-skills\u002Farxiv-search\u002Fskills\u002Farxiv-search\u002FSKILL.md",[],[],{"skillNo":57,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":19},"01M7D6M85CKKN87TA98VCQV4H7","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fyorkeccak-scientific-skills\u002Farxiv-search\u002FSkill.md",[],[],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":19},"01M7D4ZTCS4DR3CE4JY06X84T2","working-with-legacy-code","Safely change and test untested codebases using Feathers' \"Working Effectively with Legacy Code\". Use when the user mentions \"legacy code\", \"no tests\", \"untested codebase\", \"how do I test this\", \"seams\", \"characterization tests\", \"golden master\", \"sprout method\", \"afraid to change this code\", \"monster method\", \"dependency breaking\", or \"inherited a messy codebase\". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout\u002Fwrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002Fworking-with-legacy-code\u002FSKILL.md",[],[],{"skillNo":69,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":19},"01M7D4ZHZKJD2RSNGAF63V14P4","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002FSkill.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":19},"01M7D4ZB4CJBFW5JE67CPH2QRV","web-typography","Select, pair, and implement typefaces for web projects. Use when the user mentions \"font pairing\", \"which typeface\", \"line height\", \"responsive typography\", \"web font loading\", \"type hierarchy\", \"variable fonts\", \"FOUT\u002FFOIT\", \"typographic scale\", or \"the text is hard to read\". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002Fweb-typography\u002FSKILL.md",[],[],{"skillNo":81,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M7D4Z4MVYVY3NMJ18BNV6WNA","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002FSkill.md",[],[]]