[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78J4JD4RHNRJ3JM3ENCV932":3,"skill-same:01M78J4JD4RHNRJ3JM3ENCV932":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":13,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M78J4JD4RHNRJ3JM3ENCV932","nemo-mbridge-perf-moe-vlm-training",null,"Practical guidance for training MoE VLMs in Megatron Bridge. Compares FSDP and 3D-parallel approaches, using rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments.","matheussluzz\u002FZeuZ-Agent","matheussluzz","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F209255758?v=4","01M78HKHH1D1ASC9KC9BG583DH","01M78HKHHP6W7XRXQM2SBCAM3R",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-moe-vlm-training\ndescription: Practical guidance for training MoE VLMs in Megatron Bridge. Compares FSDP and 3D-parallel approaches, using rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments.\nlicense: Apache-2.0\nwhen_to_use: Training MoE VLMs, or investigating a commit that caused MoE VLM training failure or OOM; 'MoE VLM', 'multimodal MoE', 'Qwen3-VL training', 'FSDP vs 3D-parallel for VLM', 'MoE vision language model'.\n---\n\n# MoE VLM Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml\n\n## FSDP vs 3D Parallel\n\n| Approach | Strength | Best fit |\n|---|---|---|\n| FSDP | Simplest path to a working multimodal run | first bring-up, memory-first tuning, awkward PP boundaries |\n| 3D parallel | Higher ceiling after tuning | stable models with a clean PP layout and time for deeper sweeps |\n\nFor MoE VLMs, the practical workflow is usually:\n\n1. get the first reliable run with FSDP\n2. stabilize real-data input, recompute, and memory behavior\n3. move to 3D parallel only if the throughput headroom is worth the extra work\n\n## Rounded Findings From Recent VLM Runs\n\n### Qwen3-VL class models\n\nThe main patterns were consistent across the tracker:\n\n- FSDP on GB200-class systems can already reach healthy high-teens utilization\n  with a comparatively simple setup\n- B200 FSDP runs are viable, but more sensitive to recompute choice and frozen\n  vision settings\n- 3D parallel can recover to a similar or better operating point, but only after\n  tuning MBS, recompute, and the real vision path together\n\n### Real data vs mock data\n\nMock-data VLM runs are not trustworthy performance proxies. In the experiments,\nimage-free mock runs looked closer to \"roughly twice as fast\" than \"slightly\noptimistic\" when compared with real multimodal input.\n\nUse real or realistic image payloads before drawing any conclusion about VLM\nthroughput.\n\n### Smaller multimodal MoE runs\n\nThe smaller Qwen3.5-style multimodal experiments reinforce the same lessons:\n\n- HybridEP is a solid default on GB200\n- TE-scoped CUDA graphs help once the training loop is stable\n- larger MBS can pay off, but only if the vision encoder does not become the\n  next bottleneck\n\n## Decision Guide\n\n### Choose FSDP when\n\n- you are bringing up a new VLM for the first time\n- the model has awkward stage boundaries across embedding, vision, and decoder\n- memory fit matters more than absolute throughput\n- you may freeze the vision stack during decoder-focused tuning\n\n### Choose 3D parallel when\n\n- the model is already stable under FSDP\n- the PP layout is clear and repeatable\n- you can sweep MBS, recompute, and CUDA-graph scope together\n- the goal is best steady-state throughput, not easiest bring-up\n\n## Key Tuning Knobs\n\n1. **Freeze the vision stack when appropriate**: if the work is decoder-focused,\n   freezing the vision side often gives a small but real throughput gain and\n   reduces memory pressure.\n\n2. **Sweep MBS aggressively**: VLMs are more MBS-sensitive than text-only MoE\n   runs because the vision path changes the compute-to-overhead balance.\n\n3. **Prefer selective recompute once the model fits**: full recompute is a\n   useful bring-up tool, but selective recompute is usually the better steady\n   state.\n\n4. **Match CUDA-graph scope to the workload**: `attn moe_router moe_preprocess`\n   is the safer MoE default, while narrower scopes can still be useful for\n   controlled experiments.\n\n5. **Use ETP only when EP alone is insufficient**: it can unlock a layout, but\n   it also introduces more communication and more tuning surface.\n\n## Representative Config Families\n\n### FSDP-first GB200 path\n\n```text\nTP=1  CP=1  PP=1\nEP sized to the expert topology, often large\nDispatcher: HybridEP on GB200-class systems\nRecompute: start with full, then relax toward selective recompute\n```\n\n### 3D-parallel GB200 path\n\n```text\nTP=1  CP=1  PP=1 or modest PP\nEP and ETP sized to the expert topology\nDispatcher: HybridEP\nCUDA Graph: start narrow, then widen only after the real-data path is stable\n```\n\n## Compatibility\n\n| Feature | FSDP | 3D parallel |\n|---|---|---|\n| HybridEP on GB200 | strong default | strong default once topology is stable |\n| CUDA graphs | useful after bring-up | useful, but more scope-sensitive |\n| Freeze vision | natural fit | possible, but less often used as the headline perf path |\n| Selective recompute | recommended | recommended |\n\n## Pitfalls\n\n1. **Mock multimodal data is misleading**: it can make the decoder look much\n   healthier than the real end-to-end VLM path.\n\n2. **The vision encoder can dominate unexpectedly**: profile encoder, projector,\n   and decoder separately before attributing everything to the dispatcher.\n\n3. **Do not compare FSDP and 3D-parallel runs with different effective work**:\n   normalize by useful tokens and workload shape, not only by step time.\n\n4. **ETP is not free**: use it as a fit or topology tool, not as the default.\n\n5. **Recompute and CUDA-graph choices are coupled**: the setting that gets the\n   model to fit is often not the setting that gives the best steady-state speed.\n","# MoE VLM Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml\n\n## FSDP vs 3D Parallel\n\n| Approach | Strength | Best fit |\n|---|---|---|\n| FSDP | Simplest path to a working multimodal run | first bring-up, memory-first tuning, awkward PP boundaries |\n| 3D parallel | Higher ceiling after tuning | stable models with a clean PP layout and time for deeper sweeps |\n\nFor MoE VLMs, the practical workflow is usually:\n\n1. get the first reliable run with FSDP\n2. stabilize real-data input, recompute, and memory behavior\n3. move to 3D parallel only if the throughput headroom is worth the extra work\n\n## Rounded Findings From Recent VLM Runs\n\n### Qwen3-VL class models\n\nThe main patterns were consistent across the tracker:\n\n- FSDP on GB200-class systems can already reach healthy high-teens utilization\n  with a comparatively simple setup\n- B200 FSDP runs are viable, but more sensitive to recompute choice and frozen\n  vision settings\n- 3D parallel can recover to a similar or better operating point, but only after\n  tuning MBS, recompute, and the real vision path together\n\n### Real data vs mock data\n\nMock-data VLM runs are not trustworthy performance proxies. In the experiments,\nimage-free mock runs looked closer to \"roughly twice as fast\" than \"slightly\noptimistic\" when compared with real multimodal input.\n\nUse real or realistic image payloads before drawing any conclusion about VLM\nthroughput.\n\n### Smaller multimodal MoE runs\n\nThe smaller Qwen3.5-style multimodal experiments reinforce the same lessons:\n\n- HybridEP is a solid default on GB200\n- TE-scoped CUDA graphs help once the training loop is stable\n- larger MBS can pay off, but only if the vision encoder does not become the\n  next bottleneck\n\n## Decision Guide\n\n### Choose FSDP when\n\n- you are bringing up a new VLM for the first time\n- the model has awkward stage boundaries across embedding, vision, and decoder\n- memory fit matters more than absolute throughput\n- you may freeze the vision stack during decoder-focused tuning\n\n### Choose 3D parallel when\n\n- the model is already stable under FSDP\n- the PP layout is clear and repeatable\n- you can sweep MBS, recompute, and CUDA-graph scope together\n- the goal is best steady-state throughput, not easiest bring-up\n\n## Key Tuning Knobs\n\n1. **Freeze the vision stack when appropriate**: if the work is decoder-focused,\n   freezing the vision side often gives a small but real throughput gain and\n   reduces memory pressure.\n\n2. **Sweep MBS aggressively**: VLMs are more MBS-sensitive than text-only MoE\n   runs because the vision path changes the compute-to-overhead balance.\n\n3. **Prefer selective recompute once the model fits**: full recompute is a\n   useful bring-up tool, but selective recompute is usually the better steady\n   state.\n\n4. **Match CUDA-graph scope to the workload**: `attn moe_router moe_preprocess`\n   is the safer MoE default, while narrower scopes can still be useful for\n   controlled experiments.\n\n5. **Use ETP only when EP alone is insufficient**: it can unlock a layout, but\n   it also introduces more communication and more tuning surface.\n\n## Representative Config Families\n\n### FSDP-first GB200 path\n\n```text\nTP=1  CP=1  PP=1\nEP sized to the expert topology, often large\nDispatcher: HybridEP on GB200-class systems\nRecompute: start with full, then relax toward selective recompute\n```\n\n### 3D-parallel GB200 path\n\n```text\nTP=1  CP=1  PP=1 or modest PP\nEP and ETP sized to the expert topology\nDispatcher: HybridEP\nCUDA Graph: start narrow, then widen only after the real-data path is stable\n```\n\n## Compatibility\n\n| Feature | FSDP | 3D parallel |\n|---|---|---|\n| HybridEP on GB200 | strong default | strong default once topology is stable |\n| CUDA graphs | useful after bring-up | useful, but more scope-sensitive |\n| Freeze vision | natural fit | possible, but less often used as the headline perf path |\n| Selective recompute | recommended | recommended |\n\n## Pitfalls\n\n1. **Mock multimodal data is misleading**: it can make the decoder look much\n   healthier than the real end-to-end VLM path.\n\n2. **The vision encoder can dominate unexpectedly**: profile encoder, projector,\n   and decoder separately before attributing everything to the dispatcher.\n\n3. **Do not compare FSDP and 3D-parallel runs with different effective work**:\n   normalize by useful tokens and workload shape, not only by step time.\n\n4. **ETP is not free**: use it as a fit or topology tool, not as the default.\n\n5. **Recompute and CUDA-graph choices are coupled**: the setting that gets the\n   model to fit is often not the setting that gives the best steady-state speed.\n",{"name":5,"license":14,"description":7,"when_to_use":23},"Training MoE VLMs, or investigating a commit that caused MoE VLM training failure or OOM; 'MoE VLM', 'multimodal MoE', 'Qwen3-VL training', 'FSDP vs 3D-parallel for VLM', 'MoE vision language model'.","main","fc4f2c7759ae400c66cd38656db70ead882a0d08","catalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent#readme","https:\u002F\u002Fgithub.com\u002Fmatheussluzz","2026-08-27 05:21:51",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-perf-moe-vlm-training",{"tool":37,"command":38},"bunx","bunx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-perf-moe-vlm-training",{"tool":40,"command":41},"pnpm","pnpm dlx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-perf-moe-vlm-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002FSKILL.md",true,[45,51,55,59,65],{"path":46,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":49,"sourceHtmlUrl":50},"BENCHMARK.md","REFERENCE","参考文档",4355,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002FBENCHMARK.md",{"path":52,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":53,"sourceHtmlUrl":54},"card.yaml",2993,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml",{"path":56,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":57,"sourceHtmlUrl":58},"skill-card.md",3931,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fskill-card.md",{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"skill.oms.sig","OTHER","其他",4829,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fskill.oms.sig",{"path":66,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":67,"sourceHtmlUrl":68},"zeuz.manifest.yaml",172,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fzeuz.manifest.yaml",[70,78,85,92,99,106],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":18},"01M78JQ2R0QR8H2Z6662RFX804","gamma","Fixture skill gamma for discovery benchmarks.","MIT","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fgamma\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":18},"01M78JQ1KCVZ15APZG3FFF29K0","beta","Fixture skill beta for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fbeta\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":18},"01M78JQ0FJTK4Q9JKZMX9SVSEN","alpha","Fixture skill alpha for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Falpha\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":18},"01M78JPYVYPY7R8QS10KADQPWE","writing-for-agents","Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-for-agents\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":18},"01M78JPVYHB88RF5ATZVGX84RR","wizard","Generate an interactive bash wizard that walks a human through steps only they can perform. Use when provisioning infrastructure, setting up credentials or CI secrets, walking an unfamiliar third-party dashboard, or running a one-off migration or cutover. Don't invoke this for steps the agent can perform itself.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwizard\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":18},"01M78JPSSWNTZW89DWGJTX7716","wayfinder","Plan a huge chunk of work (more than one agent session can hold) as a shared map of decision tickets on your issue tracker, and resolve them one at a time until the way to the destination is clear.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwayfinder\u002FSKILL.md",[],[]]