[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7A6VTZX80C8BAND2PDMZR7G":3,"skill-same:01M7A6VTZX80C8BAND2PDMZR7G":71},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":32,"ownerDescription":33,"updatedAt":34,"install":35,"resources":48},"01M7A6VTZX80C8BAND2PDMZR7G","nemo-mbridge-perf-hierarchical-context-parallel",null,"Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.","themusebook\u002Fskills","themusebook","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F349927247?v=4","01M7A5C1RV5S3V5FF6ZACVDNRN","musebook","01M7A5C1SFC3KRZAXNN60XJ7KM",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-perf-hierarchical-context-parallel\ndescription: Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.\nlicense: Apache-2.0\nwhen_to_use: Scaling context parallelism beyond KV heads, or investigating a commit that changed CP config and caused OOM or a regression; 'hierarchical_context_parallel_sizes', 'a2a+p2p', 'hierarchical CP', 'CP beyond KV heads', 'multi-level CP'.\n---\n\n# Hierarchical Context Parallel Skill\n\nThis skill covers hierarchical context parallelism: nested context-parallel process\ngroups used by `cp_comm_type=\"a2a+p2p\"` and configured with\n`hierarchical_context_parallel_sizes`.\n\nFor what hierarchical CP is, when to use it, and the decision tree\n(`a2a+p2p` vs pure `a2a` vs `p2p`), see:\n\n- @docs\u002Ftraining\u002Fhierarchical-context-parallel.md\n- @skills\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fcard.yaml\n\n## Enablement\n\nMinimal Bridge override:\n\n```python\ncfg.model.context_parallel_size = 4\ncfg.model.cp_comm_type = \"a2a+p2p\"\ncfg.model.hierarchical_context_parallel_sizes = [2, 2]\ncfg.dist.use_decentralized_pg = False\n```\n\nRequired constraints:\n\n- `prod(hierarchical_context_parallel_sizes) == context_parallel_size`\n- `seq_length % (2 * context_parallel_size) == 0`\n- Transformer Engine `>= 1.12.0`\n\n## Code Anchors\n\nUpstream config and validation:\n\n```45:54:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Fmodel_parallel_config.py\ncontext_parallel_size: int = 1\n\"\"\"Splits network input along sequence dimension across GPU ranks.\"\"\"\n\nhierarchical_context_parallel_sizes: Optional[list[int]] = None\n\"\"\"Degrees of the hierarchical context parallelism. Users should provide a list to specify \n   the sizes for different levels. Taking the a2a+p2p cp comm type as example, it contains\n   groups of two levels, so the first value of the list indicates the group size of the a2a\n   communication type, and the second value indicates the group size of the p2p communication\n   type.\n\"\"\"\n```\n\n```428:433:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Ftraining\u002Farguments.py\nif args.hierarchical_context_parallel_sizes:\n    from numpy import prod\n    assert args.context_parallel_size == prod(args.hierarchical_context_parallel_sizes)\nif \"a2a+p2p\" in args.cp_comm_type:\n    assert args.hierarchical_context_parallel_sizes is not None, \\\n    \"--hierarchical-context-parallel-sizes must be set when a2a+p2p is used in cp comm\"\n```\n\nBridge MPU path:\n\n```613:648:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py\nparallel_state.initialize_model_parallel(\n    ...\n    context_parallel_size=model_config.context_parallel_size,\n    hierarchical_context_parallel_sizes=model_config.hierarchical_context_parallel_sizes,\n    ...\n)\n...\nreturn ProcessGroupCollection.use_mpu_process_groups()\n```\n\nBridge decentralized-PG path:\n\n```503:524:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py\npg_collection = ProcessGroupCollection(\n    ...\n    cp=cp_pg,\n    tp_cp=tp_cp_pg,\n    hcp=None,\n    ep=ep_pg,\n    ...\n)\n```\n\n## Implementation Map\n\nThe code anchors above show the config declarations and argument validation.\n\n### Validation (MCore)\n\n`TransformerConfig.__post_init__` enforces that `a2a+p2p` requires HCP sizes and the product matches CP.\n\n### Process group creation\n\n`parallel_state.initialize_model_parallel` creates hierarchical CP sub-groups\nwhen HCP sizes are provided via `create_hierarchical_groups`. Bridge currently\ngets those groups through the MPU-backed `ProcessGroupCollection`.\n\n### TE integration\n\n`TEDotProductAttention` passes the hierarchical groups to Transformer Engine\nwhen `a2a+p2p` is used. Requires **Transformer Engine >= 1.12.0**.\n\n## Pitfalls\n\n1. **Bridge HCP is MPU-only today**: If `use_decentralized_pg=True`, Bridge initializes flat CP groups and leaves HCP unset.\n2. **No checked-in Bridge recipe** currently exercises HCP directly.\n3. **Single-GPU load helpers** clear `hierarchical_context_parallel_sizes`.\n4. **Silent broken training on old stacks**: If you use `a2a+p2p` without setting `hierarchical_context_parallel_sizes`, MCore now asserts. Older versions would silently disable CP communication, so each rank attended only to its local chunk and produced artificially high throughput with broken gradients.\n5. **Product must match**: `prod(hierarchical_context_parallel_sizes)` must exactly equal `context_parallel_size`. A mismatch triggers an assertion.\n6. **Verify in logs**: Look for the process group initialization output. You should see `HIERARCHICAL_CONTEXT_PARALLEL_GROUPS` being created. If you only see `CONTEXT_PARALLEL_GROUP`, HCP is not active.\n\n## Verification\n\nNo dedicated Bridge end-to-end test exists yet for HCP (see @skills\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fcard.yaml\n`follow_up_validation`). Use the existing unit tests and log inspection instead.\n\nRun the decentralized-PG unit test to confirm the flat-CP behavior is preserved:\n\n```bash\nuv run python -m pytest tests\u002Funit_tests\u002Ftraining\u002Ftest_decentralized_pg.py -q\n```\n\nFor a manual smoke check, launch a 4-GPU run with a small recipe and\n`cp_comm_type=a2a+p2p` plus `hierarchical_context_parallel_sizes=[2,2]`:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1,2,3 uv run python -m torch.distributed.run --nproc_per_node=4 \\\n  scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe llama32_1b_pretrain_config \\\n  model.context_parallel_size=4 \\\n  model.cp_comm_type=a2a+p2p \\\n  \"model.hierarchical_context_parallel_sizes=[2,2]\" \\\n  train.train_iters=2\n```\n\nSuccess criteria:\n\n- Logs show `HIERARCHICAL_CONTEXT_PARALLEL_GROUPS` being created\n- Training completes at least one step without error\n- If you only see `CONTEXT_PARALLEL_GROUP`, HCP is not active\n","# Hierarchical Context Parallel Skill\n\nThis skill covers hierarchical context parallelism: nested context-parallel process\ngroups used by `cp_comm_type=\"a2a+p2p\"` and configured with\n`hierarchical_context_parallel_sizes`.\n\nFor what hierarchical CP is, when to use it, and the decision tree\n(`a2a+p2p` vs pure `a2a` vs `p2p`), see:\n\n- @docs\u002Ftraining\u002Fhierarchical-context-parallel.md\n- @skills\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fcard.yaml\n\n## Enablement\n\nMinimal Bridge override:\n\n```python\ncfg.model.context_parallel_size = 4\ncfg.model.cp_comm_type = \"a2a+p2p\"\ncfg.model.hierarchical_context_parallel_sizes = [2, 2]\ncfg.dist.use_decentralized_pg = False\n```\n\nRequired constraints:\n\n- `prod(hierarchical_context_parallel_sizes) == context_parallel_size`\n- `seq_length % (2 * context_parallel_size) == 0`\n- Transformer Engine `>= 1.12.0`\n\n## Code Anchors\n\nUpstream config and validation:\n\n```45:54:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Fmodel_parallel_config.py\ncontext_parallel_size: int = 1\n\"\"\"Splits network input along sequence dimension across GPU ranks.\"\"\"\n\nhierarchical_context_parallel_sizes: Optional[list[int]] = None\n\"\"\"Degrees of the hierarchical context parallelism. Users should provide a list to specify \n   the sizes for different levels. Taking the a2a+p2p cp comm type as example, it contains\n   groups of two levels, so the first value of the list indicates the group size of the a2a\n   communication type, and the second value indicates the group size of the p2p communication\n   type.\n\"\"\"\n```\n\n```428:433:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Ftraining\u002Farguments.py\nif args.hierarchical_context_parallel_sizes:\n    from numpy import prod\n    assert args.context_parallel_size == prod(args.hierarchical_context_parallel_sizes)\nif \"a2a+p2p\" in args.cp_comm_type:\n    assert args.hierarchical_context_parallel_sizes is not None, \\\n    \"--hierarchical-context-parallel-sizes must be set when a2a+p2p is used in cp comm\"\n```\n\nBridge MPU path:\n\n```613:648:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py\nparallel_state.initialize_model_parallel(\n    ...\n    context_parallel_size=model_config.context_parallel_size,\n    hierarchical_context_parallel_sizes=model_config.hierarchical_context_parallel_sizes,\n    ...\n)\n...\nreturn ProcessGroupCollection.use_mpu_process_groups()\n```\n\nBridge decentralized-PG path:\n\n```503:524:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py\npg_collection = ProcessGroupCollection(\n    ...\n    cp=cp_pg,\n    tp_cp=tp_cp_pg,\n    hcp=None,\n    ep=ep_pg,\n    ...\n)\n```\n\n## Implementation Map\n\nThe code anchors above show the config declarations and argument validation.\n\n### Validation (MCore)\n\n`TransformerConfig.__post_init__` enforces that `a2a+p2p` requires HCP sizes and the product matches CP.\n\n### Process group creation\n\n`parallel_state.initialize_model_parallel` creates hierarchical CP sub-groups\nwhen HCP sizes are provided via `create_hierarchical_groups`. Bridge currently\ngets those groups through the MPU-backed `ProcessGroupCollection`.\n\n### TE integration\n\n`TEDotProductAttention` passes the hierarchical groups to Transformer Engine\nwhen `a2a+p2p` is used. Requires **Transformer Engine >= 1.12.0**.\n\n## Pitfalls\n\n1. **Bridge HCP is MPU-only today**: If `use_decentralized_pg=True`, Bridge initializes flat CP groups and leaves HCP unset.\n2. **No checked-in Bridge recipe** currently exercises HCP directly.\n3. **Single-GPU load helpers** clear `hierarchical_context_parallel_sizes`.\n4. **Silent broken training on old stacks**: If you use `a2a+p2p` without setting `hierarchical_context_parallel_sizes`, MCore now asserts. Older versions would silently disable CP communication, so each rank attended only to its local chunk and produced artificially high throughput with broken gradients.\n5. **Product must match**: `prod(hierarchical_context_parallel_sizes)` must exactly equal `context_parallel_size`. A mismatch triggers an assertion.\n6. **Verify in logs**: Look for the process group initialization output. You should see `HIERARCHICAL_CONTEXT_PARALLEL_GROUPS` being created. If you only see `CONTEXT_PARALLEL_GROUP`, HCP is not active.\n\n## Verification\n\nNo dedicated Bridge end-to-end test exists yet for HCP (see @skills\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fcard.yaml\n`follow_up_validation`). Use the existing unit tests and log inspection instead.\n\nRun the decentralized-PG unit test to confirm the flat-CP behavior is preserved:\n\n```bash\nuv run python -m pytest tests\u002Funit_tests\u002Ftraining\u002Ftest_decentralized_pg.py -q\n```\n\nFor a manual smoke check, launch a 4-GPU run with a small recipe and\n`cp_comm_type=a2a+p2p` plus `hierarchical_context_parallel_sizes=[2,2]`:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1,2,3 uv run python -m torch.distributed.run --nproc_per_node=4 \\\n  scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe llama32_1b_pretrain_config \\\n  model.context_parallel_size=4 \\\n  model.cp_comm_type=a2a+p2p \\\n  \"model.hierarchical_context_parallel_sizes=[2,2]\" \\\n  train.train_iters=2\n```\n\nSuccess criteria:\n\n- Logs show `HIERARCHICAL_CONTEXT_PARALLEL_GROUPS` being created\n- Training completes at least one step without error\n- If you only see `CONTEXT_PARALLEL_GROUP`, HCP is not active\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Scaling context parallelism beyond KV heads, or investigating a commit that changed CP config and caused OOM or a regression; 'hierarchical_context_parallel_sizes', 'a2a+p2p', 'hierarchical CP', 'CP beyond KV heads', 'multi-level CP'.","main","1c523aba5cdf867a79f078847075492d7778092a","public\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel",1,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills","https:\u002F\u002Fskills.x402.wtf","https:\u002F\u002Fgithub.com\u002Fthemusebook","https:\u002F\u002Fmusebook.trade","The official Github of musebook","2026-11-02 23:42:26",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add themusebook\u002Fskills --skill nemo-mbridge-perf-hierarchical-context-parallel",{"tool":41,"command":42},"bunx","bunx skills add themusebook\u002Fskills --skill nemo-mbridge-perf-hierarchical-context-parallel",{"tool":44,"command":45},"pnpm","pnpm dlx skills add themusebook\u002Fskills --skill nemo-mbridge-perf-hierarchical-context-parallel","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002FSKILL.md",true,[49,55,59,63,67],{"path":50,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":53,"sourceHtmlUrl":54},"BENCHMARK.md","REFERENCE","参考文档",4459,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002FBENCHMARK.md",{"path":56,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":57,"sourceHtmlUrl":58},"card.yaml",2800,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fcard.yaml",{"path":60,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":61,"sourceHtmlUrl":62},"metadata.json",440,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fmetadata.json",{"path":64,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":65,"sourceHtmlUrl":66},"skill-card.md",3949,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fskill-card.md",{"path":68,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":69,"sourceHtmlUrl":70},"verification.json",1618,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-perf-hierarchical-context-parallel\u002Fverification.json",[72,81,88,95,102,109],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":77,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7AB34SPKVSDRRG009PM1CZH","youtube-clipper","YouTube 视频智能剪辑工具。下载视频和字幕，AI 分析生成精细章节（几分钟级别）， 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频，并生成总结文案。 使用场景：当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词：视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video","MIT","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7AB18BH6N7KR7939HRNWVMQ","viem-integration","Integrate EVM blockchains using viem. Use when user says \"read blockchain data\", \"send transaction\", \"interact with smart contract\", \"connect to Ethereum\", \"use viem\", \"use wagmi\", \"wallet integration\", \"viem setup\", or mentions blockchain\u002FEVM development with TypeScript.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fviem-integration\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7AB12GHGP48WCNXDQQ9222H","v4-security-foundations","Security-first Uniswap v4 hook development. Use when user mentions \"v4 hooks\", \"hook security\", \"PoolManager\", \"beforeSwap\", \"afterSwap\", or asks about V4 hook best practices, vulnerabilities, or audit requirements.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-security-foundations\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7AB0XM3PSR7V3Q2ZRR0Q0FB","v4-sdk-integration","App-layer SDK guide for building swap and liquidity experiences directly with the Uniswap v4 SDK. Use when user asks about \"v4 sdk\", \"uniswap v4\", \"v4 swap\", \"v4 liquidity\", \"PoolManager\", \"V4Planner\", \"StateView\", \"PositionManager\", \"pool state\", \"v4 position\", \"uniswap sdk\", or when building swap\u002Fliquidity UX directly with SDKs rather than via the Trading API.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-sdk-integration\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7AB0SGA93ASA588DJW4P9WT","v4-hook-generator","Generate Uniswap v4 hook contracts via OpenZeppelin MCP. Use when building custom swap logic, async swaps, hook-owned liquidity, custom curves, dynamic fees, MEV protection, limit orders, or oracle hooks.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-hook-generator\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7AAZYH75H4CS03WSW2EZ8DM","testing","For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Ftesting\u002FSKILL.md",[],[]]