[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78J4AHACXT50XWT03BXA0MM":3,"skill-same:01M78J4AHACXT50XWT03BXA0MM":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":13,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M78J4AHACXT50XWT03BXA0MM","nemo-mbridge-perf-moe-long-context",null,"Long-context MoE training guidance for Megatron Bridge. Covers CP sizing, selective recompute, dispatcher choices, and practical patterns from DSV3, Qwen3, and Qwen3-Next long-context experiments.","matheussluzz\u002FZeuZ-Agent","matheussluzz","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F209255758?v=4","01M78HKHH1D1ASC9KC9BG583DH","01M78HKHHP6W7XRXQM2SBCAM3R",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-moe-long-context\ndescription: Long-context MoE training guidance for Megatron Bridge. Covers CP sizing, selective recompute, dispatcher choices, and practical patterns from DSV3, Qwen3, and Qwen3-Next long-context experiments.\nlicense: Apache-2.0\nwhen_to_use: Training MoE at long sequence lengths, or investigating a commit that caused long-context MoE OOM or degraded throughput; 'long context MoE', '128k tokens', 'CP sizing for long sequences', 'selective recompute long context', 'MoE long-context OOM'.\n---\n\n# MoE Long-Context Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-long-context\u002Fcard.yaml\n\n## What Changes At Long Context\n\nOnce sequence length moves well past the 4K-class regime, attention memory and\nactivation residency become the dominant constraints. For MoE models, that\nusually means you need some combination of:\n\n- context parallelism\n- selective recompute\n- lower precision\n- CPU offload for optimizer state\n- a dispatcher and PP layout that do not waste the smaller remaining DP budget\n\n## Rounded Scaling Patterns\n\n### DSV3 on H100\n\nThe DSV3 long-context runs show a stable pattern:\n\n- selective recompute works better than full recompute once you move past the\n  shortest contexts\n- throughput stays in a fairly narrow band from mid-length through very long\n  contexts if CP is increased appropriately\n- the trade shifts from \"memory fit\" to \"GPU-count feasibility\" as CP grows\n\nIn other words, long context does not immediately collapse utilization if the\nlayout is chosen well, but it does consume the DP budget very quickly.\n\n### Qwen3-Next on GB200\n\nQwen3-Next behaves more like a memory-sensitive medium-scale model:\n\n- 8K and 32K remain practical with moderate CP\n- 64K is possible, but the throughput drop is noticeable and memory becomes\n  much tighter\n- pipeline layout and grouped-GEMM improvements matter almost as much as CP\n\n### Qwen3 235B on GB200\n\nQwen3 235B shows that long context can still be efficient on NVL72 systems when\nTP, CP, and HybridEP are coordinated. The best 128K-class configurations are\nnot just \"fit-only\" recipes; they can remain highly efficient if routing,\nparallelism, and recompute are balanced.\n\n## CP Sizing Rules Of Thumb\n\n1. **Start from a 4K shard target**: a good first guess is\n   `CP ~= seq_len \u002F 4096`, then round to a practical power-of-two layout.\n\n2. **Keep DP alive if possible**: long-context scaling becomes brittle once CP,\n   EP, TP, and PP together squeeze DP down to the floor.\n\n3. **Prefer selective recompute**: recompute modules such as `up_proj`, `norm`,\n   `moe`, `moe_act`, or `mlp` before reaching for full recompute.\n\n4. **Avoid SDPA-heavy recompute at very long context**: recomputing attention\n   internals can add a lot of work for less memory benefit than recomputing\n   smaller MoE and MLP-side modules.\n\n5. **Use TP as another lever on NVL72 systems**: GB200 and GB300 runs can\n   sometimes trade some CP for TP while still staying efficient.\n\n6. **Assume GBS will need to shrink**: as CP rises and DP falls, you may need\n   to reduce global batch size or accept higher GA.\n\n## Representative Config Families\n\n### DSV3 at 128K on H100\n\n```text\nTP=1  CP=32  EP=32  PP=8  VPP=4\nPrecision: FP8-class\nDispatcher: DeepEP\nRecompute: up_proj, norm, moe, mlp\nExtra memory help: optimizer CPU offload\n```\n\n### DSV3 at 256K on H100\n\n```text\nTP=1  CP=64  EP=32  PP=8  EDP=2  VPP=4\nPrecision: FP8-class\nDispatcher: DeepEP\nRecompute: up_proj, norm, moe, mlp\nExtra memory help: optimizer CPU offload\n```\n\n### Qwen3 235B at 128K on GB200\n\n```text\nTP=4  CP=4  EP=32  PP=4  VPP=12\nPrecision: BF16 or MXFP8\nDispatcher: HybridEP\nRecompute: moe_act, norm\nCUDA Graph: attn + moe_router + moe_preprocess\n```\n\n## Recompute And CUDA Graph Guidance\n\nFor long-context MoE training:\n\n- start with selective recompute\n- add CUDA graphs only after the shapes and routing path are stable\n- keep sequence length and MBS fixed when using CUDA graphs\n- if the run depends on highly dynamic batches, prefer eager execution\n\nUseful references:\n\n- @docs\u002Ftraining\u002Factivation-recomputation.md\n- @skills\u002Fnemo-mbridge-perf-cuda-graphs\u002FSKILL.md\n\n## Pitfalls\n\n1. **CP does not replace EP or PP**: it adds another dimension; it does not make\n   the others disappear.\n\n2. **A good 4K baseline can still be a bad long-context baseline**: routing mode,\n   recompute choice, and offload strategy often need to change.\n\n3. **GPU-count feasibility becomes the real constraint**: very long context can\n   look fine in a single recipe, then become impossible once EP and PP are added\n   honestly across the full model.\n\n4. **CUDA graphs need static shapes**: variable-length batches and opportunistic\n   padding strategies can silently break the path.\n\n5. **Container and kernel support matters more at 128K+**: long-context paths\n   tend to rely on newer kernels and bug fixes than short-context bring-up does.\n","# MoE Long-Context Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-long-context\u002Fcard.yaml\n\n## What Changes At Long Context\n\nOnce sequence length moves well past the 4K-class regime, attention memory and\nactivation residency become the dominant constraints. For MoE models, that\nusually means you need some combination of:\n\n- context parallelism\n- selective recompute\n- lower precision\n- CPU offload for optimizer state\n- a dispatcher and PP layout that do not waste the smaller remaining DP budget\n\n## Rounded Scaling Patterns\n\n### DSV3 on H100\n\nThe DSV3 long-context runs show a stable pattern:\n\n- selective recompute works better than full recompute once you move past the\n  shortest contexts\n- throughput stays in a fairly narrow band from mid-length through very long\n  contexts if CP is increased appropriately\n- the trade shifts from \"memory fit\" to \"GPU-count feasibility\" as CP grows\n\nIn other words, long context does not immediately collapse utilization if the\nlayout is chosen well, but it does consume the DP budget very quickly.\n\n### Qwen3-Next on GB200\n\nQwen3-Next behaves more like a memory-sensitive medium-scale model:\n\n- 8K and 32K remain practical with moderate CP\n- 64K is possible, but the throughput drop is noticeable and memory becomes\n  much tighter\n- pipeline layout and grouped-GEMM improvements matter almost as much as CP\n\n### Qwen3 235B on GB200\n\nQwen3 235B shows that long context can still be efficient on NVL72 systems when\nTP, CP, and HybridEP are coordinated. The best 128K-class configurations are\nnot just \"fit-only\" recipes; they can remain highly efficient if routing,\nparallelism, and recompute are balanced.\n\n## CP Sizing Rules Of Thumb\n\n1. **Start from a 4K shard target**: a good first guess is\n   `CP ~= seq_len \u002F 4096`, then round to a practical power-of-two layout.\n\n2. **Keep DP alive if possible**: long-context scaling becomes brittle once CP,\n   EP, TP, and PP together squeeze DP down to the floor.\n\n3. **Prefer selective recompute**: recompute modules such as `up_proj`, `norm`,\n   `moe`, `moe_act`, or `mlp` before reaching for full recompute.\n\n4. **Avoid SDPA-heavy recompute at very long context**: recomputing attention\n   internals can add a lot of work for less memory benefit than recomputing\n   smaller MoE and MLP-side modules.\n\n5. **Use TP as another lever on NVL72 systems**: GB200 and GB300 runs can\n   sometimes trade some CP for TP while still staying efficient.\n\n6. **Assume GBS will need to shrink**: as CP rises and DP falls, you may need\n   to reduce global batch size or accept higher GA.\n\n## Representative Config Families\n\n### DSV3 at 128K on H100\n\n```text\nTP=1  CP=32  EP=32  PP=8  VPP=4\nPrecision: FP8-class\nDispatcher: DeepEP\nRecompute: up_proj, norm, moe, mlp\nExtra memory help: optimizer CPU offload\n```\n\n### DSV3 at 256K on H100\n\n```text\nTP=1  CP=64  EP=32  PP=8  EDP=2  VPP=4\nPrecision: FP8-class\nDispatcher: DeepEP\nRecompute: up_proj, norm, moe, mlp\nExtra memory help: optimizer CPU offload\n```\n\n### Qwen3 235B at 128K on GB200\n\n```text\nTP=4  CP=4  EP=32  PP=4  VPP=12\nPrecision: BF16 or MXFP8\nDispatcher: HybridEP\nRecompute: moe_act, norm\nCUDA Graph: attn + moe_router + moe_preprocess\n```\n\n## Recompute And CUDA Graph Guidance\n\nFor long-context MoE training:\n\n- start with selective recompute\n- add CUDA graphs only after the shapes and routing path are stable\n- keep sequence length and MBS fixed when using CUDA graphs\n- if the run depends on highly dynamic batches, prefer eager execution\n\nUseful references:\n\n- @docs\u002Ftraining\u002Factivation-recomputation.md\n- @skills\u002Fnemo-mbridge-perf-cuda-graphs\u002FSKILL.md\n\n## Pitfalls\n\n1. **CP does not replace EP or PP**: it adds another dimension; it does not make\n   the others disappear.\n\n2. **A good 4K baseline can still be a bad long-context baseline**: routing mode,\n   recompute choice, and offload strategy often need to change.\n\n3. **GPU-count feasibility becomes the real constraint**: very long context can\n   look fine in a single recipe, then become impossible once EP and PP are added\n   honestly across the full model.\n\n4. **CUDA graphs need static shapes**: variable-length batches and opportunistic\n   padding strategies can silently break the path.\n\n5. **Container and kernel support matters more at 128K+**: long-context paths\n   tend to rely on newer kernels and bug fixes than short-context bring-up does.\n",{"name":5,"license":14,"description":7,"when_to_use":23},"Training MoE at long sequence lengths, or investigating a commit that caused long-context MoE OOM or degraded throughput; 'long context MoE', '128k tokens', 'CP sizing for long sequences', 'selective recompute long context', 'MoE long-context OOM'.","main","fc4f2c7759ae400c66cd38656db70ead882a0d08","catalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent#readme","https:\u002F\u002Fgithub.com\u002Fmatheussluzz","2026-08-27 05:21:51",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-perf-moe-long-context",{"tool":37,"command":38},"bunx","bunx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-perf-moe-long-context",{"tool":40,"command":41},"pnpm","pnpm dlx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-perf-moe-long-context","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002FSKILL.md",true,[45,51,55,59,65],{"path":46,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":49,"sourceHtmlUrl":50},"BENCHMARK.md","REFERENCE","参考文档",4357,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002FBENCHMARK.md",{"path":52,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":53,"sourceHtmlUrl":54},"card.yaml",3531,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002Fcard.yaml",{"path":56,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":57,"sourceHtmlUrl":58},"skill-card.md",3922,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002Fskill-card.md",{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"skill.oms.sig","OTHER","其他",4829,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002Fskill.oms.sig",{"path":66,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":67,"sourceHtmlUrl":68},"zeuz.manifest.yaml",172,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-perf-moe-long-context\u002Fzeuz.manifest.yaml",[70,78,85,92,99,106],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":18},"01M78JQ2R0QR8H2Z6662RFX804","gamma","Fixture skill gamma for discovery benchmarks.","MIT","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fgamma\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":18},"01M78JQ1KCVZ15APZG3FFF29K0","beta","Fixture skill beta for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fbeta\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":18},"01M78JQ0FJTK4Q9JKZMX9SVSEN","alpha","Fixture skill alpha for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Falpha\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":18},"01M78JPYVYPY7R8QS10KADQPWE","writing-for-agents","Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-for-agents\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":18},"01M78JPVYHB88RF5ATZVGX84RR","wizard","Generate an interactive bash wizard that walks a human through steps only they can perform. Use when provisioning infrastructure, setting up credentials or CI secrets, walking an unfamiliar third-party dashboard, or running a one-off migration or cutover. Don't invoke this for steps the agent can perform itself.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwizard\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":18},"01M78JPSSWNTZW89DWGJTX7716","wayfinder","Plan a huge chunk of work (more than one agent session can hold) as a shared map of decision tickets on your issue tracker, and resolve them one at a time until the way to the destination is clear.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwayfinder\u002FSKILL.md",[],[]]