nemo-mbridge-mlm-bridge-training
Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples.
- License 许可
- Apache-2.0
- 最近更新
- 2026-07-18
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- nemo-mbridge-mlm-bridge-training
- license
- Apache-2.0
- description
- Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples.
- when_to_use
- Running training, comparing MLM vs Bridge loss curves, translating MLM CLI args to Bridge config, or investigating why loss curves diverged after a commit; 'how do I run training', 'MLM vs Bridge', 'correlation test'.
02 资源文件 1
- 参考文档metadata.json478 B
03 SKILL.md 正文
MLM vs Bridge Training
For how they differ, the arg mapping tables, gotchas, and translation script, see:
- @docs/megatron-lm-to-megatron-bridge.md
First Answer Checklist
For MLM-vs-Bridge correlation questions, always name these items up front:
- Bridge recipe:
vanilla_gpt_pretrain_config. - Bridge entry point:
scripts/training/run_recipe.py. - MLM entry point:
3rdparty/Megatron-LM/pretrain_gpt.py. - Launch wrapper for both:
uv run python -m torch.distributed.run. - Fresh-run cleanup:
rm -rf nemo_experimentsbefore the Bridge run.
Also state that MLM needs
PYTHONPATH=3rdparty/Megatron-LM:$PYTHONPATH, matched Bridge and MLM losses
should agree within BF16 rounding, and files under 3rdparty/Megatron-LM/
should not be modified from this repo.
Correlation Testing
Use vanilla_gpt_pretrain_config for loss-correlation testing. This recipe uses
bare GPTModelProvider defaults (LayerNorm, GeLU, learned_absolute position
embeddings, vocab_size inherited from tokenizer) — matching MLM
pretrain_gpt.py defaults with no args.
MLM Correlation Run (2L/256H, 1 GPU)
PYTHONPATH=3rdparty/Megatron-LM:$PYTHONPATH \
uv run python -m torch.distributed.run --nproc_per_node=1 \
3rdparty/Megatron-LM/pretrain_gpt.py \
--num-layers 2 --hidden-size 256 --num-attention-heads 4 \
--ffn-hidden-size 1024 --seq-length 512 --max-position-embeddings 512 \
--micro-batch-size 4 --global-batch-size 32 \
--train-iters 10 --eval-iters 2 --eval-interval 10 \
--mock-data --bf16 --use-mcore-models \
--tokenizer-type NullTokenizer --vocab-size 32000 \
--lr 3e-4 --min-lr 3e-5 --seed 1234 --log-interval 1
Bridge Correlation Run (same config, 1 GPU)
rm -rf nemo_experiments && \
uv run python -m torch.distributed.run --nproc_per_node=1 \
scripts/training/run_recipe.py \
--recipe vanilla_gpt_pretrain_config \
model.num_layers=2 model.hidden_size=256 \
model.num_attention_heads=4 model.ffn_hidden_size=1024 \
model.seq_length=512 dataset.seq_length=512 \
train.train_iters=10 train.global_batch_size=32 train.micro_batch_size=4 \
validation.eval_interval=10 validation.eval_iters=2 \
optimizer.lr=3e-4 optimizer.min_lr=3e-5 \
scheduler.lr_warmup_iters=1 scheduler.lr_decay_iters=10 \
rng.seed=1234 logger.log_interval=1
Verification
With matched parameters the LM losses should be nearly identical at each
iteration. Compare lm loss values from both logs — they should agree to
within BF16 rounding.
Multi-GPU Examples
MLM 2-GPU with TP=2
PYTHONPATH=3rdparty/Megatron-LM:$PYTHONPATH \
uv run python -m torch.distributed.run --nproc_per_node=2 \
3rdparty/Megatron-LM/pretrain_gpt.py \
--tensor-model-parallel-size 2 --sequence-parallel \
--num-layers 4 --hidden-size 256 --num-attention-heads 4 \
--seq-length 1024 --max-position-embeddings 1024 \
--micro-batch-size 2 --global-batch-size 16 \
--train-iters 10 --eval-iters 2 --eval-interval 10 \
--mock-data --bf16 --use-mcore-models \
--tokenizer-type NullTokenizer --vocab-size 1024 \
--lr 1e-4 --log-interval 1
Bridge 2-GPU with TP=2
rm -rf nemo_experiments && \
uv run python -m torch.distributed.run --nproc_per_node=2 \
scripts/training/run_recipe.py \
--recipe vanilla_gpt_pretrain_config \
model.tensor_model_parallel_size=2 model.sequence_parallel=true \
model.num_layers=4 model.hidden_size=256 \
model.num_attention_heads=4 model.ffn_hidden_size=1024 \
model.seq_length=1024 dataset.seq_length=1024 \
train.train_iters=10 train.global_batch_size=16 train.micro_batch_size=2 \
validation.eval_interval=10 validation.eval_iters=2 \
scheduler.lr_warmup_iters=2 scheduler.lr_decay_iters=10 \
logger.log_interval=1
Available Recipes
Common recipes (use with --recipe):
vanilla_gpt_pretrain_config— Minimal GPT (bare GPTModelProvider defaults, ideal for correlation testing and custom configs)llama32_1b_pretrain_config— Llama 3.2 1B (16L, 2048H, GBS=512, seq=8192)llama3_8b_pretrain_config— Llama 3 8Bqwen3_8b_pretrain_config— Qwen3 8Bdeepseek_v2_lite_pretrain_config— DeepSeek-V2-Lite 16B MoE
SFT/PEFT variants use _sft_config / _peft_config suffix.
Megatron-Core Submodule
For what the submodule is and why two versions exist, see @docs/megatron-lm-to-megatron-bridge.md.
Check current version
./scripts/switch_mcore.sh status
Switch to dev for testing newer MCore features
./scripts/switch_mcore.sh dev
# uv sync (without --locked) since lockfile is for main
uv sync
Switch back to main
./scripts/switch_mcore.sh main
After pulling latest main
When you pull the latest Bridge main branch, the submodule pointer may have been updated. Re-sync the submodule:
git submodule update --init 3rdparty/Megatron-LM
Pitfalls
-
Always
rm -rf nemo_experimentsbefore a fresh correlation run. Bridge auto-resumes from stale checkpoints silently. -
uv runrequired: Always useuv run python -m torch.distributed.run(not baretorchrunorpython). -
MLM PYTHONPATH: Must include
3rdparty/Megatron-LMsogpt_builders.pyis importable. -
Scheduler overrides: When overriding
train.train_itersto a small value, also setscheduler.lr_warmup_itersandscheduler.lr_decay_itersor you get an assertion error. -
Use
dataset.seq_lengthin CLI overrides for both pretraining and fine-tuning datasets. -
MoE OOM: Large MoE models require full activation recomputation and typically multi-node EP. TP does NOT reduce per-GPU expert memory.
-
uv sync --lockedfails after switching to dev: The lockfile is generated against the main MCore commit. Useuv sync(without--locked) when on dev.
04 同仓库更多 Skills
arxiv-search
Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.
arxiv-search
Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.
working-with-legacy-code
Safely change and test untested codebases using Feathers' "Working Effectively with Legacy Code". Use when the user mentions "legacy code", "no tests", "untested codebase", "how do I test this", "seams", "characterization tests", "golden master", "sprout method", "afraid to change this code", "monster method", "dependency breaking", or "inherited a messy codebase". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout/wrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.
working-with-legacy-code
Safely change and test untested codebases using Feathers' "Working Effectively with Legacy Code". Use when the user mentions "legacy code", "no tests", "untested codebase", "how do I test this", "seams", "characterization tests", "golden master", "sprout method", "afraid to change this code", "monster method", "dependency breaking", or "inherited a messy codebase". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout/wrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.
web-typography
Select, pair, and implement typefaces for web projects. Use when the user mentions "font pairing", "which typeface", "line height", "responsive typography", "web font loading", "type hierarchy", "variable fonts", "FOUT/FOIT", "typographic scale", or "the text is hard to read". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.
web-typography
Select, pair, and implement typefaces for web projects. Use when the user mentions "font pairing", "which typeface", "line height", "responsive typography", "web font loading", "type hierarchy", "variable fonts", "FOUT/FOIT", "typographic scale", or "the text is hard to read". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.