[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79FESF00TZM8WQEPAPGASJ7":3,"skill-same:01M79FESF00TZM8WQEPAPGASJ7":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":14,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M79FESF00TZM8WQEPAPGASJ7","nemo-mbridge-perf-moe-vlm-training",null,"Practical guidance for training MoE VLMs in Megatron Bridge. Compares FSDP and 3D-parallel approaches, using rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments.","Elogic360\u002Fsoftware-factory","Elogic360","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F156535519?v=4","01M74NH3F0RB13PWB7JZDE2V80","elogic360","01M74NH3FPC65082MDZVMFHDM3",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-moe-vlm-training\ndescription: Practical guidance for training MoE VLMs in Megatron Bridge. Compares FSDP and 3D-parallel approaches, using rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments.\nlicense: Apache-2.0\nwhen_to_use: Training MoE VLMs, or investigating a commit that caused MoE VLM training failure or OOM; 'MoE VLM', 'multimodal MoE', 'Qwen3-VL training', 'FSDP vs 3D-parallel for VLM', 'MoE vision language model'.\n---\n\n# MoE VLM Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml\n\n## FSDP vs 3D Parallel\n\n| Approach | Strength | Best fit |\n|---|---|---|\n| FSDP | Simplest path to a working multimodal run | first bring-up, memory-first tuning, awkward PP boundaries |\n| 3D parallel | Higher ceiling after tuning | stable models with a clean PP layout and time for deeper sweeps |\n\nFor MoE VLMs, the practical workflow is usually:\n\n1. get the first reliable run with FSDP\n2. stabilize real-data input, recompute, and memory behavior\n3. move to 3D parallel only if the throughput headroom is worth the extra work\n\n## Rounded Findings From Recent VLM Runs\n\n### Qwen3-VL class models\n\nThe main patterns were consistent across the tracker:\n\n- FSDP on GB200-class systems can already reach healthy high-teens utilization\n  with a comparatively simple setup\n- B200 FSDP runs are viable, but more sensitive to recompute choice and frozen\n  vision settings\n- 3D parallel can recover to a similar or better operating point, but only after\n  tuning MBS, recompute, and the real vision path together\n\n### Real data vs mock data\n\nMock-data VLM runs are not trustworthy performance proxies. In the experiments,\nimage-free mock runs looked closer to \"roughly twice as fast\" than \"slightly\noptimistic\" when compared with real multimodal input.\n\nUse real or realistic image payloads before drawing any conclusion about VLM\nthroughput.\n\n### Smaller multimodal MoE runs\n\nThe smaller Qwen3.5-style multimodal experiments reinforce the same lessons:\n\n- HybridEP is a solid default on GB200\n- TE-scoped CUDA graphs help once the training loop is stable\n- larger MBS can pay off, but only if the vision encoder does not become the\n  next bottleneck\n\n## Decision Guide\n\n### Choose FSDP when\n\n- you are bringing up a new VLM for the first time\n- the model has awkward stage boundaries across embedding, vision, and decoder\n- memory fit matters more than absolute throughput\n- you may freeze the vision stack during decoder-focused tuning\n\n### Choose 3D parallel when\n\n- the model is already stable under FSDP\n- the PP layout is clear and repeatable\n- you can sweep MBS, recompute, and CUDA-graph scope together\n- the goal is best steady-state throughput, not easiest bring-up\n\n## Key Tuning Knobs\n\n1. **Freeze the vision stack when appropriate**: if the work is decoder-focused,\n   freezing the vision side often gives a small but real throughput gain and\n   reduces memory pressure.\n\n2. **Sweep MBS aggressively**: VLMs are more MBS-sensitive than text-only MoE\n   runs because the vision path changes the compute-to-overhead balance.\n\n3. **Prefer selective recompute once the model fits**: full recompute is a\n   useful bring-up tool, but selective recompute is usually the better steady\n   state.\n\n4. **Match CUDA-graph scope to the workload**: `attn moe_router moe_preprocess`\n   is the safer MoE default, while narrower scopes can still be useful for\n   controlled experiments.\n\n5. **Use ETP only when EP alone is insufficient**: it can unlock a layout, but\n   it also introduces more communication and more tuning surface.\n\n## Representative Config Families\n\n### FSDP-first GB200 path\n\n```text\nTP=1  CP=1  PP=1\nEP sized to the expert topology, often large\nDispatcher: HybridEP on GB200-class systems\nRecompute: start with full, then relax toward selective recompute\n```\n\n### 3D-parallel GB200 path\n\n```text\nTP=1  CP=1  PP=1 or modest PP\nEP and ETP sized to the expert topology\nDispatcher: HybridEP\nCUDA Graph: start narrow, then widen only after the real-data path is stable\n```\n\n## Compatibility\n\n| Feature | FSDP | 3D parallel |\n|---|---|---|\n| HybridEP on GB200 | strong default | strong default once topology is stable |\n| CUDA graphs | useful after bring-up | useful, but more scope-sensitive |\n| Freeze vision | natural fit | possible, but less often used as the headline perf path |\n| Selective recompute | recommended | recommended |\n\n## Pitfalls\n\n1. **Mock multimodal data is misleading**: it can make the decoder look much\n   healthier than the real end-to-end VLM path.\n\n2. **The vision encoder can dominate unexpectedly**: profile encoder, projector,\n   and decoder separately before attributing everything to the dispatcher.\n\n3. **Do not compare FSDP and 3D-parallel runs with different effective work**:\n   normalize by useful tokens and workload shape, not only by step time.\n\n4. **ETP is not free**: use it as a fit or topology tool, not as the default.\n\n5. **Recompute and CUDA-graph choices are coupled**: the setting that gets the\n   model to fit is often not the setting that gives the best steady-state speed.\n","# MoE VLM Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml\n\n## FSDP vs 3D Parallel\n\n| Approach | Strength | Best fit |\n|---|---|---|\n| FSDP | Simplest path to a working multimodal run | first bring-up, memory-first tuning, awkward PP boundaries |\n| 3D parallel | Higher ceiling after tuning | stable models with a clean PP layout and time for deeper sweeps |\n\nFor MoE VLMs, the practical workflow is usually:\n\n1. get the first reliable run with FSDP\n2. stabilize real-data input, recompute, and memory behavior\n3. move to 3D parallel only if the throughput headroom is worth the extra work\n\n## Rounded Findings From Recent VLM Runs\n\n### Qwen3-VL class models\n\nThe main patterns were consistent across the tracker:\n\n- FSDP on GB200-class systems can already reach healthy high-teens utilization\n  with a comparatively simple setup\n- B200 FSDP runs are viable, but more sensitive to recompute choice and frozen\n  vision settings\n- 3D parallel can recover to a similar or better operating point, but only after\n  tuning MBS, recompute, and the real vision path together\n\n### Real data vs mock data\n\nMock-data VLM runs are not trustworthy performance proxies. In the experiments,\nimage-free mock runs looked closer to \"roughly twice as fast\" than \"slightly\noptimistic\" when compared with real multimodal input.\n\nUse real or realistic image payloads before drawing any conclusion about VLM\nthroughput.\n\n### Smaller multimodal MoE runs\n\nThe smaller Qwen3.5-style multimodal experiments reinforce the same lessons:\n\n- HybridEP is a solid default on GB200\n- TE-scoped CUDA graphs help once the training loop is stable\n- larger MBS can pay off, but only if the vision encoder does not become the\n  next bottleneck\n\n## Decision Guide\n\n### Choose FSDP when\n\n- you are bringing up a new VLM for the first time\n- the model has awkward stage boundaries across embedding, vision, and decoder\n- memory fit matters more than absolute throughput\n- you may freeze the vision stack during decoder-focused tuning\n\n### Choose 3D parallel when\n\n- the model is already stable under FSDP\n- the PP layout is clear and repeatable\n- you can sweep MBS, recompute, and CUDA-graph scope together\n- the goal is best steady-state throughput, not easiest bring-up\n\n## Key Tuning Knobs\n\n1. **Freeze the vision stack when appropriate**: if the work is decoder-focused,\n   freezing the vision side often gives a small but real throughput gain and\n   reduces memory pressure.\n\n2. **Sweep MBS aggressively**: VLMs are more MBS-sensitive than text-only MoE\n   runs because the vision path changes the compute-to-overhead balance.\n\n3. **Prefer selective recompute once the model fits**: full recompute is a\n   useful bring-up tool, but selective recompute is usually the better steady\n   state.\n\n4. **Match CUDA-graph scope to the workload**: `attn moe_router moe_preprocess`\n   is the safer MoE default, while narrower scopes can still be useful for\n   controlled experiments.\n\n5. **Use ETP only when EP alone is insufficient**: it can unlock a layout, but\n   it also introduces more communication and more tuning surface.\n\n## Representative Config Families\n\n### FSDP-first GB200 path\n\n```text\nTP=1  CP=1  PP=1\nEP sized to the expert topology, often large\nDispatcher: HybridEP on GB200-class systems\nRecompute: start with full, then relax toward selective recompute\n```\n\n### 3D-parallel GB200 path\n\n```text\nTP=1  CP=1  PP=1 or modest PP\nEP and ETP sized to the expert topology\nDispatcher: HybridEP\nCUDA Graph: start narrow, then widen only after the real-data path is stable\n```\n\n## Compatibility\n\n| Feature | FSDP | 3D parallel |\n|---|---|---|\n| HybridEP on GB200 | strong default | strong default once topology is stable |\n| CUDA graphs | useful after bring-up | useful, but more scope-sensitive |\n| Freeze vision | natural fit | possible, but less often used as the headline perf path |\n| Selective recompute | recommended | recommended |\n\n## Pitfalls\n\n1. **Mock multimodal data is misleading**: it can make the decoder look much\n   healthier than the real end-to-end VLM path.\n\n2. **The vision encoder can dominate unexpectedly**: profile encoder, projector,\n   and decoder separately before attributing everything to the dispatcher.\n\n3. **Do not compare FSDP and 3D-parallel runs with different effective work**:\n   normalize by useful tokens and workload shape, not only by step time.\n\n4. **ETP is not free**: use it as a fit or topology tool, not as the default.\n\n5. **Recompute and CUDA-graph choices are coupled**: the setting that gets the\n   model to fit is often not the setting that gives the best steady-state speed.\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Training MoE VLMs, or investigating a commit that caused MoE VLM training failure or OOM; 'MoE VLM', 'multimodal MoE', 'Qwen3-VL training', 'FSDP vs 3D-parallel for VLM', 'MoE vision language model'.","main","c475fc64e94e803b50d14a791dfee8853fbb2c5c","skills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory","https:\u002F\u002Fgithub.com\u002FElogic360","2026-11-08 22:21:22",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add Elogic360\u002Fsoftware-factory --skill nemo-mbridge-perf-moe-vlm-training",{"tool":37,"command":38},"bunx","bunx skills add Elogic360\u002Fsoftware-factory --skill nemo-mbridge-perf-moe-vlm-training",{"tool":40,"command":41},"pnpm","pnpm dlx skills add Elogic360\u002Fsoftware-factory --skill nemo-mbridge-perf-moe-vlm-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002FSKILL.md",true,[45,51,55,59,63],{"path":46,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":49,"sourceHtmlUrl":50},"BENCHMARK.md","REFERENCE","参考文档",4355,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002FBENCHMARK.md",{"path":52,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":53,"sourceHtmlUrl":54},"card.yaml",2993,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml",{"path":56,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":57,"sourceHtmlUrl":58},"evals\u002Fevals.json",1770,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fevals\u002Fevals.json",{"path":60,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":61,"sourceHtmlUrl":62},"skill-card.md",3931,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fskill-card.md",{"path":64,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":67,"sourceHtmlUrl":68},"skill.oms.sig","OTHER","其他",4829,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fskill.oms.sig",[70,77,84,91,98,105],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M79G206MT3J6S0GY3C45P2KV","vss-summarize-video","Use to summarize a recorded video via the LVS summarization microservice (HITL-gated) with a VLM fallback. Not for report generation or live RTSP captioning.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-summarize-video\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M79G1WN63PGRE8CKD47WEBQ5","vss-setup-video-analytics-api","Use to deploy the vss-video-analytics-api REST service standalone (config-source, data-log bind, Elasticsearch, optional Kafka). Not for full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-video-analytics-api\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M79G1S9F7VY3EF6H53PRP5AN","vss-setup-behavior-analytics","Use to deploy the vss-behavior-analytics service standalone (entrypoint, config-source, optional calibration). Not for the full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-behavior-analytics\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M79G1N1S1Y27PFZPYMNK1MTQ","vss-search-archive","Use this skill to run top-level VSS fusion search on archived video, or to ingest video files \u002F RTSP streams for search. Do NOT use for ad-hoc visual Q&A (use vss-ask-video), live captioning (use vss-deploy-dense-captioning), or video summarization and reports (use vss-summarize-video).","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-search-archive\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":19},"01M79G1H6WN0Q6C0VCAXCZX6B7","vss-query-analytics","Use this skill when reading video-analytics metrics, incidents, alerts, and sensor data via the VA-MCP server (port 9901). Not for live VLM or incident-range narrative reports.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-query-analytics\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":19},"01M79G1DYPNXQS7QHZ0AAF5FX5","vss-manage-video-io-storage","Use to call the VIOS REST API (sensor list, timelines, clip extraction, snapshots, add\u002Fdelete sensors and streams). Not for VLM inference or search.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-manage-video-io-storage\u002FSKILL.md",[],[]]