nemo-mbridge-perf-moe-comm-overlap
MoE expert-parallel communication overlap in Megatron Bridge. Covers dispatch/combine overlap, flex dispatcher backends, and expert wgrad scheduling.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- nemo-mbridge-perf-moe-comm-overlap
- license
- Apache-2.0
- description
- MoE expert-parallel communication overlap in Megatron Bridge. Covers dispatch/combine overlap, flex dispatcher backends, and expert wgrad scheduling.
- when_to_use
- Tuning MoE communication overlap, or tracing a MoE throughput regression to a comm-overlap config change; 'overlap_moe_expert_parallel_comm', 'MoE dispatch overlap', 'flex dispatcher', 'DeepEP overlap', 'expert wgrad scheduling'.
02 资源文件 5
- 参考文档BENCHMARK.md3.9 KB
- 参考文档card.yaml2.0 KB
- 参考文档metadata.json401 B
- 参考文档skill-card.md4.0 KB
- 参考文档verification.json1.5 KB
03 SKILL.md 正文
MoE Communication Overlap
For the higher-level overview, see:
- @docs/training/communication-overlap.md
- @skills/nemo-mbridge-perf-moe-comm-overlap/card.yaml
Quick Decision
Use MoE communication overlap when:
EP > 1- token dispatch or combine time is visible in the profile
- the run is already correct and you are now tuning throughput
Avoid turning it on as an early bring-up step. It is easier to validate after the dispatcher, routing mode, and recompute plan are already stable.
Enablement
cfg.comm_overlap.overlap_moe_expert_parallel_comm = True
# Optional: delayed wgrad for additional overlap
cfg.comm_overlap.delay_wgrad_compute = True
# IMPORTANT: disable shared expert overlap when using dispatch overlap
cfg.model.moe_shared_expert_overlap = False
Prerequisites
expert_model_parallel_size > 1num_moe_experts > 1moe_token_dispatcher_typemust be"alltoall"or"flex"- Precision: BF16 or FP16
- If PP is used, VPP (
virtual_pipeline_model_parallel_size) must be set (non-None)
Flex dispatcher activation
Setting moe_flex_dispatcher_backend alone does not activate flex dispatch.
You must also set moe_token_dispatcher_type = "flex".
Recompute And CUDA Graph Interaction
- Full recompute is not a good companion for the overlap path.
delay_wgrad_computeadds further constraints if CUDA-graph scopes include attention or MoE-router work.- In practice, selective recompute is the safer pairing when overlap is enabled.
Measured Short-Run Caveat
A 2026-05-18 current-main H100 x16 smoke on Qwen3 30B-A3B mock pretraining
used EP=16, alltoall, global batch size 1024, CUDA graphs disabled, and
moe_permute_fusion=false because the PyTorch 25.11 / TE / Triton stack failed
in Transformer Engine fused permutation in prior bring-up.
Results were directional rather than release-grade:
- no EP overlap: 41.25s steady-state mean over iterations 3-8
- EP overlap: 31.31s steady-state mean over iterations 3-8
- EP overlap plus
delay_wgrad_compute: 31.20s steady-state mean over iterations 3-8
Treat this as evidence that EP overlap can help an inter-node alltoall MoE
shape when communication is exposed. It is not proof that delayed wgrad is a
separate win, and it does not validate the fused permutation path. An earlier
2026-05-16 short smoke on the same shape showed the same pattern.
Code Anchors
- Overlap validation:
src/megatron/bridge/training/comm_overlap.py - Flex dispatcher backend:
src/megatron/bridge/training/flex_dispatcher_backend.py - Config:
src/megatron/bridge/training/config.py - Unit tests:
tests/unit_tests/training/test_comm_overlap.py - DeepEP tests:
tests/unit_tests/training/test_deepep.py
Pitfalls
-
Shared expert overlap conflict:
moe_shared_expert_overlapandoverlap_moe_expert_parallel_commcan conflict. Disable shared expert overlap when using the dispatch overlap path. -
PP without VPP: MoE overlap requires VPP when pipeline parallelism is active. Without it, the overlap scheduling cannot interleave correctly.
-
Flex != backend flag:
moe_flex_dispatcher_backend="deepep"alone does nothing ifmoe_token_dispatcher_typeis still"alltoall". -
Conservative recipe defaults: Most public recipes leave MoE overlap disabled. You need to explicitly enable it via overrides.
-
Performance gains are workload-dependent: overlap helps most when dispatch communication is already a visible slice of step time. It is not guaranteed to help every small or lightly loaded EP run.
Verification
Look for overlap-related log messages during initialization. The comm overlap
validation in comm_overlap.py will raise if prerequisites are not met, so a
clean startup confirms the feature is active.
For a short performance-harness smoke, keep the command shape explicit and vary only one overlap knob at a time:
uv run python scripts/performance/run_script.py \
-m qwen \
-mr qwen3_30b_a3b \
--task pretrain \
-g h100 \
-c bf16 \
-ng 16 \
-gn 8 \
--max_steps 8 \
--cuda_graph_impl none \
--moe_flex_dispatcher_backend None \
--moe_a2a_overlap false \
--tokenizer_type NullTokenizer \
comm_overlap.overlap_moe_expert_parallel_comm=true \
comm_overlap.delay_wgrad_compute=false \
model.moe_shared_expert_overlap=false
If fused MoE permutation fails during bring-up, add
model.moe_permute_fusion=false to separate overlap timing from runtime-stack
validation, then retest with the matched production container.
04 同仓库更多 Skills
testing
For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.
qedgen
Formally verify programs by writing Lean 4 proofs. Trigger this skill whenever the user wants to formally verify code, generate Lean 4 proofs, prove properties about algorithms or smart contracts, verify invariants, convert program logic into formal specifications, or anything involving Lean 4 and formal verification. Also trigger when the user mentions "qedgen", "lean proof", "formal proof", "verify my code", "prove correctness", "formal verification", or wants mathematical guarantees about their implementation.
solana-dev
Use when user asks to "build a Solana dapp", "write an Anchor program", "create a token", "debug Solana errors", "set up wallet connection", "test my Solana program", "deploy to devnet", or "explain Solana concepts" (rent, accounts, PDAs, CPIs, etc.). Also use for quick on-chain lookups via public RPC + curl — "what's the balance of <wallet>", "look up transaction <sig>", "token balance for <account>", "check this address on mainnet/devnet". End-to-end Solana development playbook covering wallet connection, Anchor/Pinocchio programs, Codama client generation, LiteSVM/Mollusk/Surfpool testing, security checklists, and JSON-RPC curl lookups against public clusters. Integrates with the Solana MCP server for live documentation search. Prefers framework-kit (@solana/client + @solana/react-hooks) for UI, wallet-standard-first connection (incl. ConnectorKit), @solana/kit for client/RPC code, and @solana/web3-compat for legacy boundaries.
solana-common-errors
Diagnose and fix common Solana development errors — GLIBC mismatches, Anchor version conflicts, cargo build-sbf / platform-tools failures, LiteSVM issues, edition2024 crate pins, RPC airdrops, and Anchor 0.29→0.32 migrations. Use when builds fail, Anchor CLI won't install, or CI breaks on Debian/Ubuntu.
skill-creator
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
phantom-wallet-mcp
Execute wallet operations through the Phantom MCP server. Use when the user wants to interact with their Phantom wallet directly — get addresses, transfer SOL or SPL tokens, buy/swap tokens, sign transactions, and sign messages across Solana, Ethereum, Bitcoin, and Sui. Requires the @phantom/mcp-server to be configured as an MCP server.