[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78CZRYJFEVE4KDXGR0VFCRE":3,"skill-same:01M78CZRYJFEVE4KDXGR0VFCRE":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":14,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M78CZRYJFEVE4KDXGR0VFCRE","nemo-mbridge-perf-expert-parallel-overlap",null,"Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including overlap_moe_expert_parallel_comm, delay_wgrad_compute, and flex dispatcher backends such as DeepEP and HybridEP.","Solizardking\u002Fskillhub-main","Solizardking","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F108096810?v=4","01M786KDB10AYV042N0F9BD5B5","Clawd","01M786NYYXSZK1D05WM8JQPC48",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-expert-parallel-overlap\ndescription: Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including overlap_moe_expert_parallel_comm, delay_wgrad_compute, and flex dispatcher backends such as DeepEP and HybridEP.\nlicense: Apache-2.0\nwhen_to_use: Enabling EP overlap to hide dispatch\u002Fcombine latency, or tracing a throughput regression to an EP overlap config change; 'overlap_moe_expert_parallel_comm', 'delay_wgrad_compute', 'flex dispatcher', 'DeepEP overlap', 'HybridEP overlap'.\n---\n\n# MoE Expert-Parallel Overlap Skill\n\n## References\n\n- Stable docs: @docs\u002Ftraining\u002Fcommunication-overlap.md\n- Structured metadata: @skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fcard.yaml\n\n## What It Is\n\nExpert-parallel (EP) overlap hides the cost of token dispatch\u002Fcombine all-to-all\ncommunication by running it concurrently with expert FFN compute. Optionally,\ndelayed expert weight-gradient computation (`delay_wgrad_compute`) provides\nadditional overlap by deferring wgrad to overlap with the next layer's forward.\n\nBridge supports two dispatcher paths:\n\n| Dispatcher | Backend | When to use |\n|---|---|---|\n| `alltoall` | Standard MoE all-to-all | Default, broadest compatibility |\n| `flex` | DeepEP or HybridEP | Higher overlap on Ampere\u002FHopper\u002FBlackwell |\n\n## Quick Decision\n\nUse EP overlap when:\n\n- the model is MoE with `EP > 1`\n- expert dispatch\u002Fcombine communication is a meaningful part of step time\n- you have memory headroom and are tuning for throughput\n\nPrefer:\n\n- `alltoall` dispatcher for the first rollout (broader compatibility)\n- `flex` + DeepEP\u002FHybridEP when running on supported GPUs and seeking\n  additional gains\n\nAvoid EP overlap when:\n\n- full activation recompute is enabled\n- `moe_shared_expert_overlap` is enabled\n- the run is still being brought up for correctness\n- PyTorch \u003C 2.6.0\n\nExpected outcome:\n\n- if all-to-all dispatch is a clear profile bottleneck, overlap can produce a\n  modest to meaningful speedup\n- if the run is tiny, communication-light, or dominated by another wall, the\n  gain may be negligible\n\n## Correctness-First alltoall Benchmark\n\nFor the plain EP-overlap isolation benchmark, keep flex dispatch and delayed\nwgrad disabled. The measured shape was Qwen3 MoE 30B-A3B SFT on 16 H100 GPUs:\n`EP=16`, `alltoall`, BF16, global batch size 1024, CUDA graphs disabled,\n`moe_permute_fusion=false`, measured over iterations 3-8.\n\nUse these overrides for the plain-overlap case:\n\n```bash\n--cuda_graph_impl none \\\n--moe_flex_dispatcher_backend None \\\n--moe_a2a_overlap false \\\ncomm_overlap.overlap_moe_expert_parallel_comm=true \\\ncomm_overlap.delay_wgrad_compute=false \\\nmodel.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` for this isolation test: the performance\nharness helper enables both `overlap_moe_expert_parallel_comm` and\n`delay_wgrad_compute`, so it does not isolate plain EP overlap.\n\nSteady-window timing from that benchmark:\n\n| Case | Steady mean | Relative |\n|---|---:|---:|\n| no EP overlap | 41.25s | 1.000x |\n| EP overlap | 31.31s | 1.317x |\n| EP overlap plus `delay_wgrad_compute` | 31.20s | 1.322x |\n\nThis is evidence for enabling plain EP overlap on this inter-node all-to-all\nshape. It does not show a meaningful independent win from delayed wgrad, and it\ndoes not validate fused MoE permutation because that path was disabled for the\nruntime stack.\n\n## Enablement\n\n### alltoall dispatcher\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.moe_shared_expert_overlap = False\n\ncfg.model.expert_model_parallel_size = 8\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.bf16 = True\ncfg.model.fp16 = False\n```\n\nEnable `delay_wgrad_compute=True` only after the plain overlap path is known to\nwork and its extra compatibility constraints have been checked.\n\n### flex dispatcher (DeepEP or HybridEP)\n\n```python\nfrom megatron.bridge.training.flex_dispatcher_backend import apply_flex_dispatcher_backend\n\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = True\ncfg.model.moe_shared_expert_overlap = False\n\napply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"deepep\")\n# or: apply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"hybridep\")\n```\n\n## Compatibility And Constraints\n\n- `expert_model_parallel_size > 1`\n- `num_moe_experts > 1`\n- `moe_token_dispatcher_type` must be `\"alltoall\"` or `\"flex\"`\n- `moe_shared_expert_overlap = False`\n- Base precision is BF16 or FP16\n- PyTorch `>= 2.6.0`\n- If `PP > 1`, `virtual_pipeline_model_parallel_size` must be set\n- `recompute_granularity != \"full\"`, `recompute_method = None`,\n  `recompute_num_layers = None`\n- `mtp_num_layers` must be `None` or `1`\n- `delay_wgrad_compute` requires `overlap_moe_expert_parallel_comm` as a\n  prerequisite\n- `delay_wgrad_compute` with `overlap_grad_reduce` requires TE >= 2.7.0\n- `delay_wgrad_compute` with `gradient_accumulation_fusion` requires TE >= 2.7.0\n- CUDA graph `attn` scope + `delay_wgrad_compute` requires TE >= 2.12.0,\n  `gradient_accumulation_fusion = True`, and no attention bias\n- DeepEP: Ampere, Hopper, B200, B300 GPUs only\n- HybridEP: Ampere, Hopper, B200, B300, GB200\u002FGB300 with NVL72\n\n## Minimal Working Config\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.expert_model_parallel_size = 4\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.moe_shared_expert_overlap = False\ncfg.model.bf16 = True\n```\n\nUse this as the correctness-first starting point. Add delayed wgrad, flex\ndispatch, and CUDA-graph interactions only after the plain overlap path is\nknown to work.\n\n## Minimal Runnable Command\n\nPerformance harness example inside a Slurm allocation. Keep the model,\nparallelism, dispatcher, and runtime fixed, and vary only the two overlap\noverrides:\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  -gn 8 \\\n  --max_steps 8 \\\n  --cuda_graph_impl none \\\n  --moe_flex_dispatcher_backend None \\\n  --moe_a2a_overlap false \\\n  --tokenizer_type NullTokenizer \\\n  comm_overlap.overlap_moe_expert_parallel_comm=true \\\n  comm_overlap.delay_wgrad_compute=false \\\n  model.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` when separating plain EP overlap from\ndelayed wgrad: the performance harness helper enables both\n`overlap_moe_expert_parallel_comm` and `delay_wgrad_compute`.\n\nUnit test verification:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py -k \"moe\" \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n## Verification\n\n### Unit tests\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n### Log checks\n\nAfter a successful run with EP overlap:\n\n1. Confirm no assertion errors during `CommOverlapConfig` finalization\n2. Confirm `overlap_moe_expert_parallel_comm` appears as `True` in the logged\n   config\n3. If using flex dispatcher, confirm `moe_token_dispatcher_type = \"flex\"` and\n   the correct backend in logs\n\n### Success criteria\n\n- Config validation passes for the selected dispatcher and overlap settings\n- Training runs complete without hangs or assertion failures\n- Throughput improves or at least does not regress for the target workload\n- Loss trajectory matches baseline (overlap should not affect convergence)\n\n## Code Anchors\n\n### Bridge overlap validation\n\n```470:505:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.overlap_moe_expert_parallel_comm is True:\n    assert model_cfg.expert_model_parallel_size > 1, ...\n    assert model_cfg.num_moe_experts > 1, ...\n    assert model_cfg.moe_token_dispatcher_type in [\"alltoall\", \"flex\"], ...\n    assert model_cfg.bf16 or model_cfg.fp16, ...\n    assert is_torch_min_version(\"2.6.0\"), ...\n    # ... PP + VPP check, recompute checks, shared_expert_overlap check ...\n```\n\n### Delayed wgrad validation\n\n```507:557:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.delay_wgrad_compute is True:\n    # TE version checks for overlap_grad_reduce and gradient_accumulation_fusion\n    # CUDA graph scope validations for delayed wgrad\n    assert overlap_moe_expert_parallel_comm, ...\n```\n\n### Flex-dispatcher activation\n\n```27:72:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fflex_dispatcher_backend.py\ndef apply_flex_dispatcher_backend(...):\n    # GPU architecture check for DeepEP \u002F HybridEP\n    model_config.moe_token_dispatcher_type = \"flex\"\n    model_config.moe_flex_dispatcher_backend = moe_flex_dispatcher_backend\n    model_config.moe_shared_expert_overlap = False\n```\n\n### Perf harness override\n\n```149:156:scripts\u002Fperformance\u002Futils\u002Foverrides.py\ndef _set_moe_a2a_overlap_overrides(recipe, moe_a2a_overlap=False):\n    if moe_a2a_overlap:\n        recipe.comm_overlap.overlap_moe_expert_parallel_comm = True\n        recipe.comm_overlap.delay_wgrad_compute = True\n        recipe.model.moe_shared_expert_overlap = False\n```\n\n### Tests\n\n| File | Coverage |\n|---|---|\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py` | EP overlap validation, delayed wgrad, CUDA graph + wgrad interaction |\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py` | DeepEP\u002FHybridEP helper activation and GPU gating |\n\n## Failure Diagnosis\n\n| Symptom | Likely Cause | How To Confirm | Fix |\n|---|---|---|---|\n| assert `expert_model_parallel_size > 1` | EP not configured | Check `expert_model_parallel_size` | Set EP > 1 |\n| assert `moe_token_dispatcher_type` | Wrong dispatcher | Check dispatcher type | Use `\"alltoall\"` or `\"flex\"` |\n| assert on BF16\u002FFP16 | Wrong precision | Check `bf16` and `fp16` | Set `bf16 = True` |\n| hang during training | PyTorch \u003C 2.6 | Check PyTorch version | Upgrade to >= 2.6.0 |\n| assert `virtual_pipeline_model_parallel_size` | PP > 1 without VPP | Check PP and VPP config | Set VPP when PP > 1 |\n| assert `recompute_granularity` | Full recompute enabled | Check recompute settings | Disable full recompute |\n| assert `overlap_moe_expert_parallel_comm required` | delayed wgrad without EP overlap | Check `delay_wgrad_compute` without overlap | Enable EP overlap first |\n| assert `gradient_accumulation_fusion` | CUDA graph + delayed wgrad | Check graph scope + wgrad settings | Enable `gradient_accumulation_fusion` |\n| assert on attention bias | CUDA graph attn + delayed wgrad + bias | Check `add_bias_linear` \u002F `add_qkv_bias` | Disable attention bias |\n| no throughput gain from flex dispatcher | `apply_flex_dispatcher_backend` not called | Check `moe_token_dispatcher_type` in logs | Call `apply_flex_dispatcher_backend(...)` |\n| DeepEP\u002FHybridEP silently skipped | Unsupported GPU | Check warning logs | Run on Ampere\u002FHopper\u002FBlackwell |\n\n## Known Limitations\n\n- Setting `moe_flex_dispatcher_backend` alone does not activate flex dispatch —\n  you must call `apply_flex_dispatcher_backend(...)`.\n- Public recipes are often conservative and leave MoE overlap disabled by\n  default.\n- End-to-end throughput gains have not yet been measured in a controlled Bridge\n  experiment for every model family. Code validation is stronger than a single\n  universal performance claim.\n- MoE overlap and shared-expert overlap are mutually exclusive.\n- CUDA graph plus delayed wgrad is a multi-constraint path that requires\n  careful TE version and scope validation.\n","# MoE Expert-Parallel Overlap Skill\n\n## References\n\n- Stable docs: @docs\u002Ftraining\u002Fcommunication-overlap.md\n- Structured metadata: @skills\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fcard.yaml\n\n## What It Is\n\nExpert-parallel (EP) overlap hides the cost of token dispatch\u002Fcombine all-to-all\ncommunication by running it concurrently with expert FFN compute. Optionally,\ndelayed expert weight-gradient computation (`delay_wgrad_compute`) provides\nadditional overlap by deferring wgrad to overlap with the next layer's forward.\n\nBridge supports two dispatcher paths:\n\n| Dispatcher | Backend | When to use |\n|---|---|---|\n| `alltoall` | Standard MoE all-to-all | Default, broadest compatibility |\n| `flex` | DeepEP or HybridEP | Higher overlap on Ampere\u002FHopper\u002FBlackwell |\n\n## Quick Decision\n\nUse EP overlap when:\n\n- the model is MoE with `EP > 1`\n- expert dispatch\u002Fcombine communication is a meaningful part of step time\n- you have memory headroom and are tuning for throughput\n\nPrefer:\n\n- `alltoall` dispatcher for the first rollout (broader compatibility)\n- `flex` + DeepEP\u002FHybridEP when running on supported GPUs and seeking\n  additional gains\n\nAvoid EP overlap when:\n\n- full activation recompute is enabled\n- `moe_shared_expert_overlap` is enabled\n- the run is still being brought up for correctness\n- PyTorch \u003C 2.6.0\n\nExpected outcome:\n\n- if all-to-all dispatch is a clear profile bottleneck, overlap can produce a\n  modest to meaningful speedup\n- if the run is tiny, communication-light, or dominated by another wall, the\n  gain may be negligible\n\n## Correctness-First alltoall Benchmark\n\nFor the plain EP-overlap isolation benchmark, keep flex dispatch and delayed\nwgrad disabled. The measured shape was Qwen3 MoE 30B-A3B SFT on 16 H100 GPUs:\n`EP=16`, `alltoall`, BF16, global batch size 1024, CUDA graphs disabled,\n`moe_permute_fusion=false`, measured over iterations 3-8.\n\nUse these overrides for the plain-overlap case:\n\n```bash\n--cuda_graph_impl none \\\n--moe_flex_dispatcher_backend None \\\n--moe_a2a_overlap false \\\ncomm_overlap.overlap_moe_expert_parallel_comm=true \\\ncomm_overlap.delay_wgrad_compute=false \\\nmodel.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` for this isolation test: the performance\nharness helper enables both `overlap_moe_expert_parallel_comm` and\n`delay_wgrad_compute`, so it does not isolate plain EP overlap.\n\nSteady-window timing from that benchmark:\n\n| Case | Steady mean | Relative |\n|---|---:|---:|\n| no EP overlap | 41.25s | 1.000x |\n| EP overlap | 31.31s | 1.317x |\n| EP overlap plus `delay_wgrad_compute` | 31.20s | 1.322x |\n\nThis is evidence for enabling plain EP overlap on this inter-node all-to-all\nshape. It does not show a meaningful independent win from delayed wgrad, and it\ndoes not validate fused MoE permutation because that path was disabled for the\nruntime stack.\n\n## Enablement\n\n### alltoall dispatcher\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.moe_shared_expert_overlap = False\n\ncfg.model.expert_model_parallel_size = 8\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.bf16 = True\ncfg.model.fp16 = False\n```\n\nEnable `delay_wgrad_compute=True` only after the plain overlap path is known to\nwork and its extra compatibility constraints have been checked.\n\n### flex dispatcher (DeepEP or HybridEP)\n\n```python\nfrom megatron.bridge.training.flex_dispatcher_backend import apply_flex_dispatcher_backend\n\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = True\ncfg.model.moe_shared_expert_overlap = False\n\napply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"deepep\")\n# or: apply_flex_dispatcher_backend(cfg.model, moe_flex_dispatcher_backend=\"hybridep\")\n```\n\n## Compatibility And Constraints\n\n- `expert_model_parallel_size > 1`\n- `num_moe_experts > 1`\n- `moe_token_dispatcher_type` must be `\"alltoall\"` or `\"flex\"`\n- `moe_shared_expert_overlap = False`\n- Base precision is BF16 or FP16\n- PyTorch `>= 2.6.0`\n- If `PP > 1`, `virtual_pipeline_model_parallel_size` must be set\n- `recompute_granularity != \"full\"`, `recompute_method = None`,\n  `recompute_num_layers = None`\n- `mtp_num_layers` must be `None` or `1`\n- `delay_wgrad_compute` requires `overlap_moe_expert_parallel_comm` as a\n  prerequisite\n- `delay_wgrad_compute` with `overlap_grad_reduce` requires TE >= 2.7.0\n- `delay_wgrad_compute` with `gradient_accumulation_fusion` requires TE >= 2.7.0\n- CUDA graph `attn` scope + `delay_wgrad_compute` requires TE >= 2.12.0,\n  `gradient_accumulation_fusion = True`, and no attention bias\n- DeepEP: Ampere, Hopper, B200, B300 GPUs only\n- HybridEP: Ampere, Hopper, B200, B300, GB200\u002FGB300 with NVL72\n\n## Minimal Working Config\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\ncfg.comm_overlap.delay_wgrad_compute = False\ncfg.model.expert_model_parallel_size = 4\ncfg.model.num_moe_experts = 64\ncfg.model.moe_token_dispatcher_type = \"alltoall\"\ncfg.model.moe_shared_expert_overlap = False\ncfg.model.bf16 = True\n```\n\nUse this as the correctness-first starting point. Add delayed wgrad, flex\ndispatch, and CUDA-graph interactions only after the plain overlap path is\nknown to work.\n\n## Minimal Runnable Command\n\nPerformance harness example inside a Slurm allocation. Keep the model,\nparallelism, dispatcher, and runtime fixed, and vary only the two overlap\noverrides:\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  -gn 8 \\\n  --max_steps 8 \\\n  --cuda_graph_impl none \\\n  --moe_flex_dispatcher_backend None \\\n  --moe_a2a_overlap false \\\n  --tokenizer_type NullTokenizer \\\n  comm_overlap.overlap_moe_expert_parallel_comm=true \\\n  comm_overlap.delay_wgrad_compute=false \\\n  model.moe_shared_expert_overlap=false\n```\n\nDo not use `--moe_a2a_overlap true` when separating plain EP overlap from\ndelayed wgrad: the performance harness helper enables both\n`overlap_moe_expert_parallel_comm` and `delay_wgrad_compute`.\n\nUnit test verification:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py -k \"moe\" \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n## Verification\n\n### Unit tests\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py -q\n```\n\n### Log checks\n\nAfter a successful run with EP overlap:\n\n1. Confirm no assertion errors during `CommOverlapConfig` finalization\n2. Confirm `overlap_moe_expert_parallel_comm` appears as `True` in the logged\n   config\n3. If using flex dispatcher, confirm `moe_token_dispatcher_type = \"flex\"` and\n   the correct backend in logs\n\n### Success criteria\n\n- Config validation passes for the selected dispatcher and overlap settings\n- Training runs complete without hangs or assertion failures\n- Throughput improves or at least does not regress for the target workload\n- Loss trajectory matches baseline (overlap should not affect convergence)\n\n## Code Anchors\n\n### Bridge overlap validation\n\n```470:505:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.overlap_moe_expert_parallel_comm is True:\n    assert model_cfg.expert_model_parallel_size > 1, ...\n    assert model_cfg.num_moe_experts > 1, ...\n    assert model_cfg.moe_token_dispatcher_type in [\"alltoall\", \"flex\"], ...\n    assert model_cfg.bf16 or model_cfg.fp16, ...\n    assert is_torch_min_version(\"2.6.0\"), ...\n    # ... PP + VPP check, recompute checks, shared_expert_overlap check ...\n```\n\n### Delayed wgrad validation\n\n```507:557:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\nif self.user_comm_overlap_cfg.delay_wgrad_compute is True:\n    # TE version checks for overlap_grad_reduce and gradient_accumulation_fusion\n    # CUDA graph scope validations for delayed wgrad\n    assert overlap_moe_expert_parallel_comm, ...\n```\n\n### Flex-dispatcher activation\n\n```27:72:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fflex_dispatcher_backend.py\ndef apply_flex_dispatcher_backend(...):\n    # GPU architecture check for DeepEP \u002F HybridEP\n    model_config.moe_token_dispatcher_type = \"flex\"\n    model_config.moe_flex_dispatcher_backend = moe_flex_dispatcher_backend\n    model_config.moe_shared_expert_overlap = False\n```\n\n### Perf harness override\n\n```149:156:scripts\u002Fperformance\u002Futils\u002Foverrides.py\ndef _set_moe_a2a_overlap_overrides(recipe, moe_a2a_overlap=False):\n    if moe_a2a_overlap:\n        recipe.comm_overlap.overlap_moe_expert_parallel_comm = True\n        recipe.comm_overlap.delay_wgrad_compute = True\n        recipe.model.moe_shared_expert_overlap = False\n```\n\n### Tests\n\n| File | Coverage |\n|---|---|\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py` | EP overlap validation, delayed wgrad, CUDA graph + wgrad interaction |\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py` | DeepEP\u002FHybridEP helper activation and GPU gating |\n\n## Failure Diagnosis\n\n| Symptom | Likely Cause | How To Confirm | Fix |\n|---|---|---|---|\n| assert `expert_model_parallel_size > 1` | EP not configured | Check `expert_model_parallel_size` | Set EP > 1 |\n| assert `moe_token_dispatcher_type` | Wrong dispatcher | Check dispatcher type | Use `\"alltoall\"` or `\"flex\"` |\n| assert on BF16\u002FFP16 | Wrong precision | Check `bf16` and `fp16` | Set `bf16 = True` |\n| hang during training | PyTorch \u003C 2.6 | Check PyTorch version | Upgrade to >= 2.6.0 |\n| assert `virtual_pipeline_model_parallel_size` | PP > 1 without VPP | Check PP and VPP config | Set VPP when PP > 1 |\n| assert `recompute_granularity` | Full recompute enabled | Check recompute settings | Disable full recompute |\n| assert `overlap_moe_expert_parallel_comm required` | delayed wgrad without EP overlap | Check `delay_wgrad_compute` without overlap | Enable EP overlap first |\n| assert `gradient_accumulation_fusion` | CUDA graph + delayed wgrad | Check graph scope + wgrad settings | Enable `gradient_accumulation_fusion` |\n| assert on attention bias | CUDA graph attn + delayed wgrad + bias | Check `add_bias_linear` \u002F `add_qkv_bias` | Disable attention bias |\n| no throughput gain from flex dispatcher | `apply_flex_dispatcher_backend` not called | Check `moe_token_dispatcher_type` in logs | Call `apply_flex_dispatcher_backend(...)` |\n| DeepEP\u002FHybridEP silently skipped | Unsupported GPU | Check warning logs | Run on Ampere\u002FHopper\u002FBlackwell |\n\n## Known Limitations\n\n- Setting `moe_flex_dispatcher_backend` alone does not activate flex dispatch —\n  you must call `apply_flex_dispatcher_backend(...)`.\n- Public recipes are often conservative and leave MoE overlap disabled by\n  default.\n- End-to-end throughput gains have not yet been measured in a controlled Bridge\n  experiment for every model family. Code validation is stronger than a single\n  universal performance claim.\n- MoE overlap and shared-expert overlap are mutually exclusive.\n- CUDA graph plus delayed wgrad is a multi-constraint path that requires\n  careful TE version and scope validation.\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Enabling EP overlap to hide dispatch\u002Fcombine latency, or tracing a throughput regression to an EP overlap config change; 'overlap_moe_expert_parallel_comm', 'delay_wgrad_compute', 'flex dispatcher', 'DeepEP overlap', 'HybridEP overlap'.","main","be3de0244dc1a57e323d8160787ebb40fb64930e","skills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main","https:\u002F\u002Fskills.x402.wtf","https:\u002F\u002Fgithub.com\u002FSolizardking","https:\u002F\u002Fmusebook.trade","Got a mile high and never came down","2026-10-08 23:20:13",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add Solizardking\u002Fskillhub-main --skill nemo-mbridge-perf-expert-parallel-overlap",{"tool":40,"command":41},"bunx","bunx skills add Solizardking\u002Fskillhub-main --skill nemo-mbridge-perf-expert-parallel-overlap",{"tool":43,"command":44},"pnpm","pnpm dlx skills add Solizardking\u002Fskillhub-main --skill nemo-mbridge-perf-expert-parallel-overlap","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002FSKILL.md",true,[48,54,58,62,66],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"BENCHMARK.md","REFERENCE","参考文档",4054,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002FBENCHMARK.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"card.yaml",7519,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fcard.yaml",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"evals\u002Fevals.json",2093,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fevals\u002Fevals.json",{"path":63,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":64,"sourceHtmlUrl":65},"skill-card.md",4375,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fskill-card.md",{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"skill.oms.sig","OTHER","其他",4837,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-expert-parallel-overlap\u002Fskill.oms.sig",[73,81,88,95,102,110],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M78E0TGCAMNPS207BA3G07S7","zkrouter","Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.","MIT","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzkrouter\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M78E0R1Z8BDT6EDTHY17MTNB","zk","For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzk\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M78E0P3ZTZ1YWF8MD5MM23QE","zk-omni-messaging","Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzk-omni-messaging\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M78E0JXZ2PNB1896VXAGDBWD","zerion-wallet-data","Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzerion-wallet-data\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":106,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":19},"01M78E0G14FH9GRRTQYMKD5E59","youtube-clipper","YouTube 视频智能剪辑工具。下载视频和字幕，AI 分析生成精细章节（几分钟级别）， 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频，并生成总结文案。 使用场景：当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词：视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M78E0C2ZYA99CFFDNB7ZMK5S","yield-optimization","Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit\u002Fwithdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter\u002Fexit a position, or check portfolio balances.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyield-optimization\u002FSKILL.md",[],[]]