[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7880BKRY5CSK76HYXDAMA9P":3,"skill-same:01M7880BKRY5CSK76HYXDAMA9P":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":14,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7880BKRY5CSK76HYXDAMA9P","nemo-mbridge-perf-moe-comm-overlap",null,"MoE expert-parallel communication overlap in Megatron Bridge. Covers dispatch\u002Fcombine overlap, flex dispatcher backends, and expert wgrad scheduling.","Solizardking\u002Ftrain2earn","Solizardking","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F108096810?v=4","01M786KDB10AYV042N0F9BD5B5","Clawd","01M786KDC5C3NPE4W44C5641EZ",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-moe-comm-overlap\ndescription: MoE expert-parallel communication overlap in Megatron Bridge. Covers dispatch\u002Fcombine overlap, flex dispatcher backends, and expert wgrad scheduling.\nlicense: Apache-2.0\nwhen_to_use: Tuning MoE communication overlap, or tracing a MoE throughput regression to a comm-overlap config change; 'overlap_moe_expert_parallel_comm', 'MoE dispatch overlap', 'flex dispatcher', 'DeepEP overlap', 'expert wgrad scheduling'.\n---\n\n# MoE Communication Overlap\n\nFor the higher-level overview, see:\n\n- @docs\u002Ftraining\u002Fcommunication-overlap.md\n- @skills\u002Fnemo-mbridge-perf-moe-comm-overlap\u002Fcard.yaml\n\n## Quick Decision\n\nUse MoE communication overlap when:\n\n- `EP > 1`\n- token dispatch or combine time is visible in the profile\n- the run is already correct and you are now tuning throughput\n\nAvoid turning it on as an early bring-up step. It is easier to validate after\nthe dispatcher, routing mode, and recompute plan are already stable.\n\n## Enablement\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\n\n# Optional: delayed wgrad for additional overlap\ncfg.comm_overlap.delay_wgrad_compute = True\n\n# IMPORTANT: disable shared expert overlap when using dispatch overlap\ncfg.model.moe_shared_expert_overlap = False\n```\n\n### Prerequisites\n\n- `expert_model_parallel_size > 1`\n- `num_moe_experts > 1`\n- `moe_token_dispatcher_type` must be `\"alltoall\"` or `\"flex\"`\n- Precision: BF16 or FP16\n- If PP is used, VPP (`virtual_pipeline_model_parallel_size`) must be set (non-`None`)\n\n### Flex dispatcher activation\n\nSetting `moe_flex_dispatcher_backend` alone does **not** activate flex dispatch.\nYou must also set `moe_token_dispatcher_type = \"flex\"`.\n\n## Recompute And CUDA Graph Interaction\n\n- Full recompute is not a good companion for the overlap path.\n- `delay_wgrad_compute` adds further constraints if CUDA-graph scopes include\n  attention or MoE-router work.\n- In practice, selective recompute is the safer pairing when overlap is enabled.\n\n## Measured Short-Run Caveat\n\nA 2026-05-18 current-main H100 x16 smoke on Qwen3 30B-A3B mock pretraining\nused `EP=16`, `alltoall`, global batch size 1024, CUDA graphs disabled, and\n`moe_permute_fusion=false` because the PyTorch 25.11 \u002F TE \u002F Triton stack failed\nin Transformer Engine fused permutation in prior bring-up.\n\nResults were directional rather than release-grade:\n\n- no EP overlap: 41.25s steady-state mean over iterations 3-8\n- EP overlap: 31.31s steady-state mean over iterations 3-8\n- EP overlap plus `delay_wgrad_compute`: 31.20s steady-state mean over\n  iterations 3-8\n\nTreat this as evidence that EP overlap can help an inter-node `alltoall` MoE\nshape when communication is exposed. It is not proof that delayed wgrad is a\nseparate win, and it does not validate the fused permutation path. An earlier\n2026-05-16 short smoke on the same shape showed the same pattern.\n\n## Code Anchors\n\n- Overlap validation: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py`\n- Flex dispatcher backend: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fflex_dispatcher_backend.py`\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py`\n- Unit tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py`\n- DeepEP tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py`\n\n## Pitfalls\n\n1. **Shared expert overlap conflict**: `moe_shared_expert_overlap` and\n   `overlap_moe_expert_parallel_comm` can conflict. Disable shared expert\n   overlap when using the dispatch overlap path.\n\n2. **PP without VPP**: MoE overlap requires VPP when pipeline parallelism is\n   active. Without it, the overlap scheduling cannot interleave correctly.\n\n3. **Flex != backend flag**: `moe_flex_dispatcher_backend=\"deepep\"` alone\n   does nothing if `moe_token_dispatcher_type` is still `\"alltoall\"`.\n\n4. **Conservative recipe defaults**: Most public recipes leave MoE overlap\n   disabled. You need to explicitly enable it via overrides.\n\n5. **Performance gains are workload-dependent**: overlap helps most when dispatch\n   communication is already a visible slice of step time. It is not guaranteed\n   to help every small or lightly loaded EP run.\n\n## Verification\n\nLook for overlap-related log messages during initialization. The comm overlap\nvalidation in `comm_overlap.py` will raise if prerequisites are not met, so a\nclean startup confirms the feature is active.\n\nFor a short performance-harness smoke, keep the command shape explicit and vary\nonly one overlap knob at a time:\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  -gn 8 \\\n  --max_steps 8 \\\n  --cuda_graph_impl none \\\n  --moe_flex_dispatcher_backend None \\\n  --moe_a2a_overlap false \\\n  --tokenizer_type NullTokenizer \\\n  comm_overlap.overlap_moe_expert_parallel_comm=true \\\n  comm_overlap.delay_wgrad_compute=false \\\n  model.moe_shared_expert_overlap=false\n```\n\nIf fused MoE permutation fails during bring-up, add\n`model.moe_permute_fusion=false` to separate overlap timing from runtime-stack\nvalidation, then retest with the matched production container.\n","# MoE Communication Overlap\n\nFor the higher-level overview, see:\n\n- @docs\u002Ftraining\u002Fcommunication-overlap.md\n- @skills\u002Fnemo-mbridge-perf-moe-comm-overlap\u002Fcard.yaml\n\n## Quick Decision\n\nUse MoE communication overlap when:\n\n- `EP > 1`\n- token dispatch or combine time is visible in the profile\n- the run is already correct and you are now tuning throughput\n\nAvoid turning it on as an early bring-up step. It is easier to validate after\nthe dispatcher, routing mode, and recompute plan are already stable.\n\n## Enablement\n\n```python\ncfg.comm_overlap.overlap_moe_expert_parallel_comm = True\n\n# Optional: delayed wgrad for additional overlap\ncfg.comm_overlap.delay_wgrad_compute = True\n\n# IMPORTANT: disable shared expert overlap when using dispatch overlap\ncfg.model.moe_shared_expert_overlap = False\n```\n\n### Prerequisites\n\n- `expert_model_parallel_size > 1`\n- `num_moe_experts > 1`\n- `moe_token_dispatcher_type` must be `\"alltoall\"` or `\"flex\"`\n- Precision: BF16 or FP16\n- If PP is used, VPP (`virtual_pipeline_model_parallel_size`) must be set (non-`None`)\n\n### Flex dispatcher activation\n\nSetting `moe_flex_dispatcher_backend` alone does **not** activate flex dispatch.\nYou must also set `moe_token_dispatcher_type = \"flex\"`.\n\n## Recompute And CUDA Graph Interaction\n\n- Full recompute is not a good companion for the overlap path.\n- `delay_wgrad_compute` adds further constraints if CUDA-graph scopes include\n  attention or MoE-router work.\n- In practice, selective recompute is the safer pairing when overlap is enabled.\n\n## Measured Short-Run Caveat\n\nA 2026-05-18 current-main H100 x16 smoke on Qwen3 30B-A3B mock pretraining\nused `EP=16`, `alltoall`, global batch size 1024, CUDA graphs disabled, and\n`moe_permute_fusion=false` because the PyTorch 25.11 \u002F TE \u002F Triton stack failed\nin Transformer Engine fused permutation in prior bring-up.\n\nResults were directional rather than release-grade:\n\n- no EP overlap: 41.25s steady-state mean over iterations 3-8\n- EP overlap: 31.31s steady-state mean over iterations 3-8\n- EP overlap plus `delay_wgrad_compute`: 31.20s steady-state mean over\n  iterations 3-8\n\nTreat this as evidence that EP overlap can help an inter-node `alltoall` MoE\nshape when communication is exposed. It is not proof that delayed wgrad is a\nseparate win, and it does not validate the fused permutation path. An earlier\n2026-05-16 short smoke on the same shape showed the same pattern.\n\n## Code Anchors\n\n- Overlap validation: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py`\n- Flex dispatcher backend: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fflex_dispatcher_backend.py`\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py`\n- Unit tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py`\n- DeepEP tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_deepep.py`\n\n## Pitfalls\n\n1. **Shared expert overlap conflict**: `moe_shared_expert_overlap` and\n   `overlap_moe_expert_parallel_comm` can conflict. Disable shared expert\n   overlap when using the dispatch overlap path.\n\n2. **PP without VPP**: MoE overlap requires VPP when pipeline parallelism is\n   active. Without it, the overlap scheduling cannot interleave correctly.\n\n3. **Flex != backend flag**: `moe_flex_dispatcher_backend=\"deepep\"` alone\n   does nothing if `moe_token_dispatcher_type` is still `\"alltoall\"`.\n\n4. **Conservative recipe defaults**: Most public recipes leave MoE overlap\n   disabled. You need to explicitly enable it via overrides.\n\n5. **Performance gains are workload-dependent**: overlap helps most when dispatch\n   communication is already a visible slice of step time. It is not guaranteed\n   to help every small or lightly loaded EP run.\n\n## Verification\n\nLook for overlap-related log messages during initialization. The comm overlap\nvalidation in `comm_overlap.py` will raise if prerequisites are not met, so a\nclean startup confirms the feature is active.\n\nFor a short performance-harness smoke, keep the command shape explicit and vary\nonly one overlap knob at a time:\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  -gn 8 \\\n  --max_steps 8 \\\n  --cuda_graph_impl none \\\n  --moe_flex_dispatcher_backend None \\\n  --moe_a2a_overlap false \\\n  --tokenizer_type NullTokenizer \\\n  comm_overlap.overlap_moe_expert_parallel_comm=true \\\n  comm_overlap.delay_wgrad_compute=false \\\n  model.moe_shared_expert_overlap=false\n```\n\nIf fused MoE permutation fails during bring-up, add\n`model.moe_permute_fusion=false` to separate overlap timing from runtime-stack\nvalidation, then retest with the matched production container.\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Tuning MoE communication overlap, or tracing a MoE throughput regression to a comm-overlap config change; 'overlap_moe_expert_parallel_comm', 'MoE dispatch overlap', 'flex dispatcher', 'DeepEP overlap', 'expert wgrad scheduling'.","main","86cc27ffd16af6cb85948f1155578e6f3f61cc4c","skillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn","https:\u002F\u002Fgithub.com\u002FSolizardking","https:\u002F\u002Fmusebook.trade","Got a mile high and never came down","2026-10-08 11:52:53",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add Solizardking\u002Ftrain2earn --skill nemo-mbridge-perf-moe-comm-overlap",{"tool":39,"command":40},"bunx","bunx skills add Solizardking\u002Ftrain2earn --skill nemo-mbridge-perf-moe-comm-overlap",{"tool":42,"command":43},"pnpm","pnpm dlx skills add Solizardking\u002Ftrain2earn --skill nemo-mbridge-perf-moe-comm-overlap","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002FSKILL.md",true,[47,53,57,61,65],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"BENCHMARK.md","REFERENCE","参考文档",4014,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002FBENCHMARK.md",{"path":54,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":55,"sourceHtmlUrl":56},"card.yaml",2075,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002Fcard.yaml",{"path":58,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":59,"sourceHtmlUrl":60},"metadata.json",401,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002Fmetadata.json",{"path":62,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":63,"sourceHtmlUrl":64},"skill-card.md",4140,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002Fskill-card.md",{"path":66,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":67,"sourceHtmlUrl":68},"verification.json",1554,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-comm-overlap\u002Fverification.json",[70,78,86,94,101,109],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M78B94RSCAKMH1QS79VE6DG9","testing","For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.","MIT","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Ftesting\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":74,"licenseFileUrl":82,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M78B81BKY8RX12HETPG8YKWK","qedgen","Formally verify programs by writing Lean 4 proofs. Trigger this skill whenever the user wants to formally verify code, generate Lean 4 proofs, prove properties about algorithms or smart contracts, verify invariants, convert program logic into formal specifications, or anything involving Lean 4 and formal verification. Also trigger when the user mentions \"qedgen\", \"lean proof\", \"formal proof\", \"verify my code\", \"prove correctness\", \"formal verification\", or wants mathematical guarantees about their implementation.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-formal-verification\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-formal-verification\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":74,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M78B7Q7C5K54E01DN83A60KG","solana-dev","Use when user asks to \"build a Solana dapp\", \"write an Anchor program\", \"create a token\", \"debug Solana errors\", \"set up wallet connection\", \"test my Solana program\", \"deploy to devnet\", or \"explain Solana concepts\" (rent, accounts, PDAs, CPIs, etc.). Also use for quick on-chain lookups via public RPC + curl — \"what's the balance of \u003Cwallet>\", \"look up transaction \u003Csig>\", \"token balance for \u003Caccount>\", \"check this address on mainnet\u002Fdevnet\". End-to-end Solana development playbook covering wallet connection, Anchor\u002FPinocchio programs, Codama client generation, LiteSVM\u002FMollusk\u002FSurfpool testing, security checklists, and JSON-RPC curl lookups against public clusters. Integrates with the Solana MCP server for live documentation search. Prefers framework-kit (@solana\u002Fclient + @solana\u002Freact-hooks) for UI, wallet-standard-first connection (incl. ConnectorKit), @solana\u002Fkit for client\u002FRPC code, and @solana\u002Fweb3-compat for legacy boundaries.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-dev\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-dev\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M78B7HXD6N6GWKSVRQ8F6Q31","solana-common-errors","Diagnose and fix common Solana development errors — GLIBC mismatches, Anchor version conflicts, cargo build-sbf \u002F platform-tools failures, LiteSVM issues, edition2024 crate pins, RPC airdrops, and Anchor 0.29→0.32 migrations. Use when builds fail, Anchor CLI won't install, or CI breaks on Debian\u002FUbuntu.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-common-errors\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":105,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M78B6T7YSYWQ0N4DP40QDXEZ","skill-creator","Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fskill-creator\u002Flicense.txt","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fskill-creator\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M78B1VFT9X5AND7AE1HZPZGF","phantom-wallet-mcp","Execute wallet operations through the Phantom MCP server. Use when the user wants to interact with their Phantom wallet directly — get addresses, transfer SOL or SPL tokens, buy\u002Fswap tokens, sign transactions, and sign messages across Solana, Ethereum, Bitcoin, and Sui. Requires the @phantom\u002Fmcp-server to be configured as an MCP server.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fphantom-wallet-mcp\u002FSKILL.md",[],[]]