[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71NX08AF05WDKF73ZS9W98E":3,"skill-same:01M71NX08AF05WDKF73ZS9W98E":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":51,"install":52,"resources":65},"01M71NX08AF05WDKF73ZS9W98E","nemo-mbridge-resiliency",null,"Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.","pattanation\u002FNorvik","pattanation","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235419268?v=4","01M71M5THHYYB15ME8R2E4FCZG","Patta","01M71M5TJM1NDM992M8PDKYR2K",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"nemo mbridge resiliency","NeMo","state machine","automatic","detection","Preemption","features","run","process","state","tolerance","commit","machine","training","re","bridge","NVIDIA","megatron","broken","fault tolerance","2026-11-10 10:56:23",false,"---\nname: nemo-mbridge-resiliency\ndescription: Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.\nlicense: Apache-2.0\nwhen_to_use: Enabling resiliency features, or investigating a commit that caused training hangs, straggler detection failures, or broken restarts; 'fault tolerance', 'straggler detection', 'hang detection', 'automatic restart', 'in-process restart', 'preemption', 'nvidia-resiliency-ext'.\n---\n\n# Resiliency\n\nStable docs: @docs\u002Ftraining\u002Fresiliency.md, @docs\u002Ftraining\u002Fcheckpointing.md\nCard: @skills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml\n\n## Enablement\n\n### Fault tolerance (Slurm only)\n\n#### Option 1: NeMo Run plugin (recommended)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import FaultTolerancePlugin\nimport nemo_run as run\n\ntask = run.Script(...)\nrun_plugins = [\n    FaultTolerancePlugin(\n        enable_ft_package=True,\n        calc_ft_timeouts=True,\n        num_in_job_restarts=3,\n        num_job_retries_on_failure=2,\n        initial_rank_heartbeat_timeout=1800,\n        rank_heartbeat_timeout=300,\n    )\n]\nrun.run(task, plugins=run_plugins, executor=executor)\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `num_in_job_restarts` | 3 | Max restarts within same job |\n| `num_job_retries_on_failure` | 2 | Max new job launches on failure |\n| `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) |\n| `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) |\n\n#### Option 2: Direct config + ft_launcher\n\n```python\nfrom megatron.bridge.training.config import FaultToleranceConfig\n\ncfg.ft = FaultToleranceConfig(\n    enable_ft_package=True,\n    calc_ft_timeouts=True,\n    simulate_fault=False,\n    simulated_fault_type=\"random\",\n)\n```\n\nLaunch with `ft_launcher` (not `torchrun`):\n\n```bash\nexport GROUP_RANK=0  # required for non-Slurm\nft_launcher \\\n    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \\\n    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \\\n    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \\\n    --ft-rank_out_of_section_timeout=300 \\\n    your_training_script.py\n```\n\n| Config parameter | Default | Description |\n|---|---|---|\n| `enable_ft_package` | False | Enable fault tolerance |\n| `calc_ft_timeouts` | False | Auto-compute optimal timeouts |\n| `simulate_fault` | False | Enable fault simulation for testing |\n| `simulated_fault_type` | `\"random\"` | `\"rank_hung\"`, `\"rank_killed\"`, or `\"random\"` |\n| `simulated_fault_rank` | None | Specific rank to fault (random if None) |\n| `simulated_fault_base_delay` | 0 | Base delay before simulating fault |\n\nSection-based timeout monitoring covers setup, training steps, checkpointing,\nand out-of-section time independently. Timeouts are saved to `ft_state.json`\nfor subsequent runs when `calc_ft_timeouts=True`.\n\n### NVRx straggler detection\n\n```python\nfrom megatron.bridge.training.config import NVRxStragglerDetectionConfig\n\ncfg.nvrx_straggler = NVRxStragglerDetectionConfig(\n    enabled=True,\n    report_time_interval=300.0,\n    calc_relative_gpu_perf=True,\n    calc_individual_gpu_perf=True,\n    num_gpu_perf_scores_to_print=5,\n    gpu_relative_perf_threshold=0.7,\n    gpu_individual_perf_threshold=0.7,\n    stop_if_detected=False,\n    enable_logging=True,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable straggler detection |\n| `report_time_interval` | 300.0 | Seconds between straggler checks |\n| `calc_relative_gpu_perf` | True | Compare ranks against each other |\n| `calc_individual_gpu_perf` | True | Track per-rank degradation over time |\n| `gpu_relative_perf_threshold` | 0.7 | Threshold for relative performance (0-1) |\n| `gpu_individual_perf_threshold` | 0.7 | Threshold for individual performance (0-1) |\n| `stop_if_detected` | False | Terminate training on straggler |\n| `num_gpu_perf_scores_to_print` | 5 | Number of best\u002Fworst scores to print |\n| `profiling_interval` | 1 | Profiling interval for detector |\n\n### Preemption\n\n#### Plugin (Slurm)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import PreemptionPlugin\n\nplugins = [\n    PreemptionPlugin(\n        preempt_time=60,\n        enable_exit_handler=True,\n        enable_exit_handler_for_data_loader=False,\n    )\n]\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `preempt_time` | 60 | Seconds before job limit to send signal |\n| `enable_exit_handler` | True | Enable signal handler in training |\n| `enable_exit_handler_for_data_loader` | False | Enable for dataloader workers |\n\n#### Direct config\n\n```python\nimport signal\ncfg.train.exit_signal_handler = True\ncfg.train.exit_signal = signal.SIGTERM\ncfg.train.exit_signal_handler_for_dataloader = False\n```\n\n### Re-run state machine (experimental)\n\n```python\nfrom megatron.bridge.training.config import RerunStateMachineConfig\n\ncfg.rerun_state_machine = RerunStateMachineConfig(\n    rerun_mode=\"validate_results\",\n    check_for_nan_in_loss=True,\n    check_for_spiky_loss=False,\n    spiky_loss_factor=10.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `rerun_mode` | `\"disabled\"` | `\"disabled\"`, `\"validate_results\"`, `\"report_determinism_stats\"` |\n| `check_for_nan_in_loss` | True | Check for NaN in loss |\n| `check_for_spiky_loss` | False | Check for unexpectedly large loss |\n| `spiky_loss_factor` | 10.0 | Loss flagged if > factor * max observed (increase for large models) |\n\nExit codes: 16 = resume to disambiguate, 17 = failed validation.\n\n### In-process restart (experimental)\n\n```python\nfrom megatron.bridge.training.config import InProcessRestartConfig\n\ncfg.inprocess_restart = InProcessRestartConfig(\n    enabled=True,\n    granularity=\"node\",\n    soft_timeout=60.0,\n    hard_timeout=90.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable in-process restart |\n| `active_world_size` | None | Ranks executing workload (rest are warm reserves) |\n| `granularity` | `\"node\"` | `\"node\"` or `\"rank\"` restart granularity |\n| `max_iterations` | None | Max restart attempts (None = unlimited) |\n| `soft_timeout` | 60.0 | Detect GIL-released hangs (seconds) |\n| `hard_timeout` | 90.0 | Force-terminate hung ranks (seconds) |\n| `heartbeat_interval` | 30.0 | Heartbeat interval (seconds) |\n| `heartbeat_timeout` | 60.0 | Missing heartbeat timeout (seconds) |\n| `barrier_timeout` | 120.0 | Distributed barrier timeout (seconds) |\n| `completion_timeout` | 120.0 | Completion barrier timeout (seconds) |\n| `empty_cuda_cache` | True | Clear CUDA cache during restart |\n| `max_rank_faults` | None | Max rank faults before terminating |\n| `monitor_process_logdir` | None | Directory for monitor logs |\n\nRequired environment variables:\n\n```bash\nexport TORCH_CPP_LOG_LEVEL=error\nexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0\nexport NCCL_NVLS_ENABLE=0\n```\n\nThe PyTorch NCCL watchdog timeout must exceed `hard_timeout`. NeMo-Run's\nSlurm Executor is not supported; launch directly with `srun --kill-on-bad-exit=0`.\n\n### Async checkpoint save\n\n```python\ncfg.checkpoint.async_save = True\ncfg.checkpoint.ckpt_format = \"torch_dist\"\n```\n\n### Local checkpointing (NVRx)\n\n```python\ncfg.checkpoint.non_persistent_local_ckpt_dir = \"\u002Flocal\u002Fscratch\u002Fckpt\"\ncfg.checkpoint.non_persistent_local_ckpt_algo = \"fully_parallel\"\n```\n\n## Code Anchors\n\n### Fault tolerance\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `FaultToleranceConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ffault_tolerance.py`\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `FaultTolerancePlugin`\n- Perf plugin: `scripts\u002Fperformance\u002Fnemo-mbridge-resiliency_plugins.py`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_fault_tolerance.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002F`\n\n### Straggler detection\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `NVRxStragglerDetectionConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fnvrx_straggler.py`\n- Train loop: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py` — `check_nvrx_straggler_detection`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002F`\n\n### In-process restart\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `InProcessRestartConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finprocess_restart.py`\n- Entry point: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fpretrain.py` — `maybe_wrap_for_inprocess_restart`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_inprocess_restart.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py`\n\n### Preemption\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `PreemptionPlugin`\n- Signal handler: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Futils\u002Fsig_utils.py`\n- Tests: `tests\u002Funit_tests\u002Frecipes\u002Ftest_run_plugins.py`\n\n### Re-run state machine\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `RerunStateMachineConfig`\n- Init: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py` — `init_rerun_state`\n\n### Checkpointing\n- Async save: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `schedule_async_save`\n- Local ckpt: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `LocalCheckpointManager`\n- Tests: `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_local_checkpointing.py`\n\n## Pitfalls\n\n1. **ft_launcher, not torchrun**: Direct `FaultToleranceConfig` requires\n   `ft_launcher`. Using `torchrun` silently disables FT. For non-Slurm,\n   set `GROUP_RANK=0`.\n\n2. **Async save requires torch_dist**: `async_save=True` only works with\n   `ckpt_format=\"torch_dist\"`. Other formats silently fail or error.\n\n3. **IPR + NeMo-Run**: In-process restart is not compatible with NeMo-Run\n   or Slurm preemption plugins. Requires specific PyTorch\u002FNCCL versions\n   and env vars.\n\n4. **NVRx vs legacy straggler**: Two detectors exist. Use NVRx\n   (`nvrx_straggler`); do not enable both.\n\n5. **stop_if_detected default**: NVRx logs but does not stop training by\n   default. Set `stop_if_detected=True` for automatic termination.\n\n6. **NCCL watchdog vs hard_timeout**: For IPR, NCCL watchdog timeout must\n   exceed `hard_timeout` or PyTorch kills the process before recovery.\n\n7. **Rerun state machine is alpha**: Use `check_for_nan_in_loss=True` for\n   NaN detection, but don't rely on full rerun workflows yet.\n\n## Verification\n\n### Fault tolerance\n```bash\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh --simulate-fault\n```\nLook for `[FaultTolerance]` \u002F `[RankMonitorServer]` log lines with section\ntimeouts. Simulated fault should trigger restart from checkpoint.\n\n### Straggler detection\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n    examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002Fstraggler_detection_example.py\n```\nLook for `GPU relative performance` and `GPU individual performance` reports\nwith per-rank scores.\n\n### Async checkpoint\nLook for `Scheduling async checkpoint save` in logs. Training iterations\nshould continue while checkpoint files are being written.\n\n### In-process restart\n```bash\npytest tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py -v\n```\nRequires compatible PyTorch\u002FNCCL versions.\n","# Resiliency\n\nStable docs: @docs\u002Ftraining\u002Fresiliency.md, @docs\u002Ftraining\u002Fcheckpointing.md\nCard: @skills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml\n\n## Enablement\n\n### Fault tolerance (Slurm only)\n\n#### Option 1: NeMo Run plugin (recommended)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import FaultTolerancePlugin\nimport nemo_run as run\n\ntask = run.Script(...)\nrun_plugins = [\n    FaultTolerancePlugin(\n        enable_ft_package=True,\n        calc_ft_timeouts=True,\n        num_in_job_restarts=3,\n        num_job_retries_on_failure=2,\n        initial_rank_heartbeat_timeout=1800,\n        rank_heartbeat_timeout=300,\n    )\n]\nrun.run(task, plugins=run_plugins, executor=executor)\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `num_in_job_restarts` | 3 | Max restarts within same job |\n| `num_job_retries_on_failure` | 2 | Max new job launches on failure |\n| `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) |\n| `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) |\n\n#### Option 2: Direct config + ft_launcher\n\n```python\nfrom megatron.bridge.training.config import FaultToleranceConfig\n\ncfg.ft = FaultToleranceConfig(\n    enable_ft_package=True,\n    calc_ft_timeouts=True,\n    simulate_fault=False,\n    simulated_fault_type=\"random\",\n)\n```\n\nLaunch with `ft_launcher` (not `torchrun`):\n\n```bash\nexport GROUP_RANK=0  # required for non-Slurm\nft_launcher \\\n    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \\\n    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \\\n    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \\\n    --ft-rank_out_of_section_timeout=300 \\\n    your_training_script.py\n```\n\n| Config parameter | Default | Description |\n|---|---|---|\n| `enable_ft_package` | False | Enable fault tolerance |\n| `calc_ft_timeouts` | False | Auto-compute optimal timeouts |\n| `simulate_fault` | False | Enable fault simulation for testing |\n| `simulated_fault_type` | `\"random\"` | `\"rank_hung\"`, `\"rank_killed\"`, or `\"random\"` |\n| `simulated_fault_rank` | None | Specific rank to fault (random if None) |\n| `simulated_fault_base_delay` | 0 | Base delay before simulating fault |\n\nSection-based timeout monitoring covers setup, training steps, checkpointing,\nand out-of-section time independently. Timeouts are saved to `ft_state.json`\nfor subsequent runs when `calc_ft_timeouts=True`.\n\n### NVRx straggler detection\n\n```python\nfrom megatron.bridge.training.config import NVRxStragglerDetectionConfig\n\ncfg.nvrx_straggler = NVRxStragglerDetectionConfig(\n    enabled=True,\n    report_time_interval=300.0,\n    calc_relative_gpu_perf=True,\n    calc_individual_gpu_perf=True,\n    num_gpu_perf_scores_to_print=5,\n    gpu_relative_perf_threshold=0.7,\n    gpu_individual_perf_threshold=0.7,\n    stop_if_detected=False,\n    enable_logging=True,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable straggler detection |\n| `report_time_interval` | 300.0 | Seconds between straggler checks |\n| `calc_relative_gpu_perf` | True | Compare ranks against each other |\n| `calc_individual_gpu_perf` | True | Track per-rank degradation over time |\n| `gpu_relative_perf_threshold` | 0.7 | Threshold for relative performance (0-1) |\n| `gpu_individual_perf_threshold` | 0.7 | Threshold for individual performance (0-1) |\n| `stop_if_detected` | False | Terminate training on straggler |\n| `num_gpu_perf_scores_to_print` | 5 | Number of best\u002Fworst scores to print |\n| `profiling_interval` | 1 | Profiling interval for detector |\n\n### Preemption\n\n#### Plugin (Slurm)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import PreemptionPlugin\n\nplugins = [\n    PreemptionPlugin(\n        preempt_time=60,\n        enable_exit_handler=True,\n        enable_exit_handler_for_data_loader=False,\n    )\n]\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `preempt_time` | 60 | Seconds before job limit to send signal |\n| `enable_exit_handler` | True | Enable signal handler in training |\n| `enable_exit_handler_for_data_loader` | False | Enable for dataloader workers |\n\n#### Direct config\n\n```python\nimport signal\ncfg.train.exit_signal_handler = True\ncfg.train.exit_signal = signal.SIGTERM\ncfg.train.exit_signal_handler_for_dataloader = False\n```\n\n### Re-run state machine (experimental)\n\n```python\nfrom megatron.bridge.training.config import RerunStateMachineConfig\n\ncfg.rerun_state_machine = RerunStateMachineConfig(\n    rerun_mode=\"validate_results\",\n    check_for_nan_in_loss=True,\n    check_for_spiky_loss=False,\n    spiky_loss_factor=10.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `rerun_mode` | `\"disabled\"` | `\"disabled\"`, `\"validate_results\"`, `\"report_determinism_stats\"` |\n| `check_for_nan_in_loss` | True | Check for NaN in loss |\n| `check_for_spiky_loss` | False | Check for unexpectedly large loss |\n| `spiky_loss_factor` | 10.0 | Loss flagged if > factor * max observed (increase for large models) |\n\nExit codes: 16 = resume to disambiguate, 17 = failed validation.\n\n### In-process restart (experimental)\n\n```python\nfrom megatron.bridge.training.config import InProcessRestartConfig\n\ncfg.inprocess_restart = InProcessRestartConfig(\n    enabled=True,\n    granularity=\"node\",\n    soft_timeout=60.0,\n    hard_timeout=90.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable in-process restart |\n| `active_world_size` | None | Ranks executing workload (rest are warm reserves) |\n| `granularity` | `\"node\"` | `\"node\"` or `\"rank\"` restart granularity |\n| `max_iterations` | None | Max restart attempts (None = unlimited) |\n| `soft_timeout` | 60.0 | Detect GIL-released hangs (seconds) |\n| `hard_timeout` | 90.0 | Force-terminate hung ranks (seconds) |\n| `heartbeat_interval` | 30.0 | Heartbeat interval (seconds) |\n| `heartbeat_timeout` | 60.0 | Missing heartbeat timeout (seconds) |\n| `barrier_timeout` | 120.0 | Distributed barrier timeout (seconds) |\n| `completion_timeout` | 120.0 | Completion barrier timeout (seconds) |\n| `empty_cuda_cache` | True | Clear CUDA cache during restart |\n| `max_rank_faults` | None | Max rank faults before terminating |\n| `monitor_process_logdir` | None | Directory for monitor logs |\n\nRequired environment variables:\n\n```bash\nexport TORCH_CPP_LOG_LEVEL=error\nexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0\nexport NCCL_NVLS_ENABLE=0\n```\n\nThe PyTorch NCCL watchdog timeout must exceed `hard_timeout`. NeMo-Run's\nSlurm Executor is not supported; launch directly with `srun --kill-on-bad-exit=0`.\n\n### Async checkpoint save\n\n```python\ncfg.checkpoint.async_save = True\ncfg.checkpoint.ckpt_format = \"torch_dist\"\n```\n\n### Local checkpointing (NVRx)\n\n```python\ncfg.checkpoint.non_persistent_local_ckpt_dir = \"\u002Flocal\u002Fscratch\u002Fckpt\"\ncfg.checkpoint.non_persistent_local_ckpt_algo = \"fully_parallel\"\n```\n\n## Code Anchors\n\n### Fault tolerance\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `FaultToleranceConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ffault_tolerance.py`\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `FaultTolerancePlugin`\n- Perf plugin: `scripts\u002Fperformance\u002Fnemo-mbridge-resiliency_plugins.py`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_fault_tolerance.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002F`\n\n### Straggler detection\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `NVRxStragglerDetectionConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fnvrx_straggler.py`\n- Train loop: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py` — `check_nvrx_straggler_detection`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002F`\n\n### In-process restart\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `InProcessRestartConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finprocess_restart.py`\n- Entry point: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fpretrain.py` — `maybe_wrap_for_inprocess_restart`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_inprocess_restart.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py`\n\n### Preemption\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `PreemptionPlugin`\n- Signal handler: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Futils\u002Fsig_utils.py`\n- Tests: `tests\u002Funit_tests\u002Frecipes\u002Ftest_run_plugins.py`\n\n### Re-run state machine\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `RerunStateMachineConfig`\n- Init: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py` — `init_rerun_state`\n\n### Checkpointing\n- Async save: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `schedule_async_save`\n- Local ckpt: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `LocalCheckpointManager`\n- Tests: `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_local_checkpointing.py`\n\n## Pitfalls\n\n1. **ft_launcher, not torchrun**: Direct `FaultToleranceConfig` requires\n   `ft_launcher`. Using `torchrun` silently disables FT. For non-Slurm,\n   set `GROUP_RANK=0`.\n\n2. **Async save requires torch_dist**: `async_save=True` only works with\n   `ckpt_format=\"torch_dist\"`. Other formats silently fail or error.\n\n3. **IPR + NeMo-Run**: In-process restart is not compatible with NeMo-Run\n   or Slurm preemption plugins. Requires specific PyTorch\u002FNCCL versions\n   and env vars.\n\n4. **NVRx vs legacy straggler**: Two detectors exist. Use NVRx\n   (`nvrx_straggler`); do not enable both.\n\n5. **stop_if_detected default**: NVRx logs but does not stop training by\n   default. Set `stop_if_detected=True` for automatic termination.\n\n6. **NCCL watchdog vs hard_timeout**: For IPR, NCCL watchdog timeout must\n   exceed `hard_timeout` or PyTorch kills the process before recovery.\n\n7. **Rerun state machine is alpha**: Use `check_for_nan_in_loss=True` for\n   NaN detection, but don't rely on full rerun workflows yet.\n\n## Verification\n\n### Fault tolerance\n```bash\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh --simulate-fault\n```\nLook for `[FaultTolerance]` \u002F `[RankMonitorServer]` log lines with section\ntimeouts. Simulated fault should trigger restart from checkpoint.\n\n### Straggler detection\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n    examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002Fstraggler_detection_example.py\n```\nLook for `GPU relative performance` and `GPU individual performance` reports\nwith per-rank scores.\n\n### Async checkpoint\nLook for `Scheduling async checkpoint save` in logs. Training iterations\nshould continue while checkpoint files are being written.\n\n### In-process restart\n```bash\npytest tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py -v\n```\nRequires compatible PyTorch\u002FNCCL versions.\n",{"name":5,"license":15,"description":7,"when_to_use":44},"Enabling resiliency features, or investigating a commit that caused training hangs, straggler detection failures, or broken restarts; 'fault tolerance', 'straggler detection', 'hang detection', 'automatic restart', 'in-process restart', 'preemption', 'nvidia-resiliency-ext'.","claude\u002Finstall-gsd-build-foQdV","d86cd127f527124b5fd0fd0bc679a4d79b7d01d7",".agents\u002Fskills\u002Fnemo-mbridge-resiliency","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik","https:\u002F\u002Fnorvik-orpin.vercel.app","https:\u002F\u002Fgithub.com\u002Fpattanation","2026-07-17 11:17:04",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add pattanation\u002FNorvik --skill nemo-mbridge-resiliency",{"tool":58,"command":59},"bunx","bunx skills add pattanation\u002FNorvik --skill nemo-mbridge-resiliency",{"tool":61,"command":62},"pnpm","pnpm dlx skills add pattanation\u002FNorvik --skill nemo-mbridge-resiliency","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002FSKILL.md",true,[66,72,76,80,84],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"BENCHMARK.md","REFERENCE","参考文档",4267,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002FBENCHMARK.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"card.yaml",4910,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"evals\u002Fevals.json",1310,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fevals\u002Fevals.json",{"path":81,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":82,"sourceHtmlUrl":83},"skill-card.md",3896,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fskill-card.md",{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"skill.oms.sig","OTHER","其他",4809,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fskill.oms.sig",[91,125,150,173,195,210],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":106,"publishedAt":124},"01M71QDV4DM8T19HEHWATEF1K4","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[98,103],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000009","前端开发",[107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123],"webapp","testing","webapp testing","for","applications","local","frontend","debugging","toolkit","logs","browser","with","playwright","web","behavior","ui","screenshots","2026-11-10 13:02:02",{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":129,"sourceHtmlUrl":130,"categories":131,"keywords":133,"publishedAt":124},"01M71QDH2GGMXGA9105C9W1N7B","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[132],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[134,135,136,120,137,138,139,110,140,141,113,142,143,144,145,146,147,28,148,149],"builder","artifacts builder","artifacts","web artifacts builder","single-file","css","Tailwind","ui components","single","use","file","components","react","multi","routing","state management",{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":154,"licenseFileUrl":155,"sourceHtmlUrl":156,"categories":157,"keywords":159,"publishedAt":124},"01M71QD9T2XC3EWMQNN9ACNWQ1","vueuse-functions","Apply VueUse composables where appropriate to build concise, maintainable Vue.js \u002F Nuxt features.","MIT","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvueuse-functions\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvueuse-functions\u002FSKILL.md",[158],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[160,161,162,163,164,165,166,167,168,169,25,170,171,172],"functions","vueuse","vueuse functions","project","nuxt","apply","github","concise","build","https","vue","js","composables",{"skillNo":174,"name":175,"nameZh":6,"description":176,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":154,"licenseFileUrl":177,"sourceHtmlUrl":178,"categories":179,"keywords":182,"publishedAt":124},"01M71QD51HWSGYMCMB7RW7PGK8","vue-testing-best-practices","Use for Vue.js testing. Covers Vitest, Vue Test Utils, component testing, mocking, testing patterns, and Playwright for E2E testing.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-testing-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-testing-best-practices\u002FSKILL.md",[180,181],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[183,108,184,170,185,119,186,187,188,166,189,110,190,191,192,143,193,171,194],"vue testing best practices","testing best practices","best practices","test","testing patterns","component","vitest","mocking","e2e","utils","patterns","e2e testing",{"skillNo":196,"name":197,"nameZh":6,"description":198,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":154,"licenseFileUrl":199,"sourceHtmlUrl":200,"categories":201,"keywords":203,"publishedAt":124},"01M71QD358H2DJ9THJRBP77T3T","vue-router-best-practices","Vue Router 4 patterns, navigation guards, route params, and route-component lifecycle interactions.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-router-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-router-best-practices\u002FSKILL.md",[202],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[204,170,205,185,206,188,207,166,193,208,209],"router","vue router best practices","route","guards","navigation","lifecycle",{"skillNo":211,"name":212,"nameZh":6,"description":213,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":154,"licenseFileUrl":214,"sourceHtmlUrl":215,"categories":216,"keywords":218,"publishedAt":124},"01M71QCYPNWDE2MQZ1TF3ED5S4","vue-best-practices","MUST be used for Vue.js tasks. Strongly recommends Composition API with `\u003Cscript setup>` and TypeScript as the standard approach. Covers Vue 3, SSR, Volar, vue-tsc. Load for any Vue, .vue files, Vue Router, Pinia, or Vite with Vue work. ALWAYS use Composition API unless the project explicitly requires Options API.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-best-practices\u002FSKILL.md",[217],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[219,170,185,220,163,221,222,223,204,224,225,226,227,228,229,110,230,143,231,232],"vue best practices","Composition","files","tasks","api","pinia","work","options","ssr","typescript","as","vite","load","standard"]