[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78A6J1K3KPAAYMA13CF1AMM":3,"skill-same:01M78A6J1K3KPAAYMA13CF1AMM":70},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":14,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M78A6J1K3KPAAYMA13CF1AMM","nemo-mbridge-resiliency",null,"Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.","Solizardking\u002Fskillhub-main","Solizardking","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F108096810?v=4","01M786KDB10AYV042N0F9BD5B5","Clawd","01M786NYYXSZK1D05WM8JQPC48",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-resiliency\ndescription: Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.\nlicense: Apache-2.0\nwhen_to_use: Enabling resiliency features, or investigating a commit that caused training hangs, straggler detection failures, or broken restarts; 'fault tolerance', 'straggler detection', 'hang detection', 'automatic restart', 'in-process restart', 'preemption', 'nvidia-resiliency-ext'.\n---\n\n# Resiliency\n\nStable docs: @docs\u002Ftraining\u002Fresiliency.md, @docs\u002Ftraining\u002Fcheckpointing.md\nCard: @skills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml\n\n## Enablement\n\n### Fault tolerance (Slurm only)\n\n#### Option 1: NeMo Run plugin (recommended)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import FaultTolerancePlugin\nimport nemo_run as run\n\ntask = run.Script(...)\nrun_plugins = [\n    FaultTolerancePlugin(\n        enable_ft_package=True,\n        calc_ft_timeouts=True,\n        num_in_job_restarts=3,\n        num_job_retries_on_failure=2,\n        initial_rank_heartbeat_timeout=1800,\n        rank_heartbeat_timeout=300,\n    )\n]\nrun.run(task, plugins=run_plugins, executor=executor)\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `num_in_job_restarts` | 3 | Max restarts within same job |\n| `num_job_retries_on_failure` | 2 | Max new job launches on failure |\n| `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) |\n| `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) |\n\n#### Option 2: Direct config + ft_launcher\n\n```python\nfrom megatron.bridge.training.config import FaultToleranceConfig\n\ncfg.ft = FaultToleranceConfig(\n    enable_ft_package=True,\n    calc_ft_timeouts=True,\n    simulate_fault=False,\n    simulated_fault_type=\"random\",\n)\n```\n\nLaunch with `ft_launcher` (not `torchrun`):\n\n```bash\nexport GROUP_RANK=0  # required for non-Slurm\nft_launcher \\\n    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \\\n    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \\\n    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \\\n    --ft-rank_out_of_section_timeout=300 \\\n    your_training_script.py\n```\n\n| Config parameter | Default | Description |\n|---|---|---|\n| `enable_ft_package` | False | Enable fault tolerance |\n| `calc_ft_timeouts` | False | Auto-compute optimal timeouts |\n| `simulate_fault` | False | Enable fault simulation for testing |\n| `simulated_fault_type` | `\"random\"` | `\"rank_hung\"`, `\"rank_killed\"`, or `\"random\"` |\n| `simulated_fault_rank` | None | Specific rank to fault (random if None) |\n| `simulated_fault_base_delay` | 0 | Base delay before simulating fault |\n\nSection-based timeout monitoring covers setup, training steps, checkpointing,\nand out-of-section time independently. Timeouts are saved to `ft_state.json`\nfor subsequent runs when `calc_ft_timeouts=True`.\n\n### NVRx straggler detection\n\n```python\nfrom megatron.bridge.training.config import NVRxStragglerDetectionConfig\n\ncfg.nvrx_straggler = NVRxStragglerDetectionConfig(\n    enabled=True,\n    report_time_interval=300.0,\n    calc_relative_gpu_perf=True,\n    calc_individual_gpu_perf=True,\n    num_gpu_perf_scores_to_print=5,\n    gpu_relative_perf_threshold=0.7,\n    gpu_individual_perf_threshold=0.7,\n    stop_if_detected=False,\n    enable_logging=True,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable straggler detection |\n| `report_time_interval` | 300.0 | Seconds between straggler checks |\n| `calc_relative_gpu_perf` | True | Compare ranks against each other |\n| `calc_individual_gpu_perf` | True | Track per-rank degradation over time |\n| `gpu_relative_perf_threshold` | 0.7 | Threshold for relative performance (0-1) |\n| `gpu_individual_perf_threshold` | 0.7 | Threshold for individual performance (0-1) |\n| `stop_if_detected` | False | Terminate training on straggler |\n| `num_gpu_perf_scores_to_print` | 5 | Number of best\u002Fworst scores to print |\n| `profiling_interval` | 1 | Profiling interval for detector |\n\n### Preemption\n\n#### Plugin (Slurm)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import PreemptionPlugin\n\nplugins = [\n    PreemptionPlugin(\n        preempt_time=60,\n        enable_exit_handler=True,\n        enable_exit_handler_for_data_loader=False,\n    )\n]\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `preempt_time` | 60 | Seconds before job limit to send signal |\n| `enable_exit_handler` | True | Enable signal handler in training |\n| `enable_exit_handler_for_data_loader` | False | Enable for dataloader workers |\n\n#### Direct config\n\n```python\nimport signal\ncfg.train.exit_signal_handler = True\ncfg.train.exit_signal = signal.SIGTERM\ncfg.train.exit_signal_handler_for_dataloader = False\n```\n\n### Re-run state machine (experimental)\n\n```python\nfrom megatron.bridge.training.config import RerunStateMachineConfig\n\ncfg.rerun_state_machine = RerunStateMachineConfig(\n    rerun_mode=\"validate_results\",\n    check_for_nan_in_loss=True,\n    check_for_spiky_loss=False,\n    spiky_loss_factor=10.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `rerun_mode` | `\"disabled\"` | `\"disabled\"`, `\"validate_results\"`, `\"report_determinism_stats\"` |\n| `check_for_nan_in_loss` | True | Check for NaN in loss |\n| `check_for_spiky_loss` | False | Check for unexpectedly large loss |\n| `spiky_loss_factor` | 10.0 | Loss flagged if > factor * max observed (increase for large models) |\n\nExit codes: 16 = resume to disambiguate, 17 = failed validation.\n\n### In-process restart (experimental)\n\n```python\nfrom megatron.bridge.training.config import InProcessRestartConfig\n\ncfg.inprocess_restart = InProcessRestartConfig(\n    enabled=True,\n    granularity=\"node\",\n    soft_timeout=60.0,\n    hard_timeout=90.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable in-process restart |\n| `active_world_size` | None | Ranks executing workload (rest are warm reserves) |\n| `granularity` | `\"node\"` | `\"node\"` or `\"rank\"` restart granularity |\n| `max_iterations` | None | Max restart attempts (None = unlimited) |\n| `soft_timeout` | 60.0 | Detect GIL-released hangs (seconds) |\n| `hard_timeout` | 90.0 | Force-terminate hung ranks (seconds) |\n| `heartbeat_interval` | 30.0 | Heartbeat interval (seconds) |\n| `heartbeat_timeout` | 60.0 | Missing heartbeat timeout (seconds) |\n| `barrier_timeout` | 120.0 | Distributed barrier timeout (seconds) |\n| `completion_timeout` | 120.0 | Completion barrier timeout (seconds) |\n| `empty_cuda_cache` | True | Clear CUDA cache during restart |\n| `max_rank_faults` | None | Max rank faults before terminating |\n| `monitor_process_logdir` | None | Directory for monitor logs |\n\nRequired environment variables:\n\n```bash\nexport TORCH_CPP_LOG_LEVEL=error\nexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0\nexport NCCL_NVLS_ENABLE=0\n```\n\nThe PyTorch NCCL watchdog timeout must exceed `hard_timeout`. NeMo-Run's\nSlurm Executor is not supported; launch directly with `srun --kill-on-bad-exit=0`.\n\n### Async checkpoint save\n\n```python\ncfg.checkpoint.async_save = True\ncfg.checkpoint.ckpt_format = \"torch_dist\"\n```\n\n### Local checkpointing (NVRx)\n\n```python\ncfg.checkpoint.non_persistent_local_ckpt_dir = \"\u002Flocal\u002Fscratch\u002Fckpt\"\ncfg.checkpoint.non_persistent_local_ckpt_algo = \"fully_parallel\"\n```\n\n## Code Anchors\n\n### Fault tolerance\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `FaultToleranceConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ffault_tolerance.py`\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `FaultTolerancePlugin`\n- Perf plugin: `scripts\u002Fperformance\u002Fnemo-mbridge-resiliency_plugins.py`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_fault_tolerance.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002F`\n\n### Straggler detection\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `NVRxStragglerDetectionConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fnvrx_straggler.py`\n- Train loop: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py` — `check_nvrx_straggler_detection`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002F`\n\n### In-process restart\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `InProcessRestartConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finprocess_restart.py`\n- Entry point: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fpretrain.py` — `maybe_wrap_for_inprocess_restart`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_inprocess_restart.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py`\n\n### Preemption\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `PreemptionPlugin`\n- Signal handler: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Futils\u002Fsig_utils.py`\n- Tests: `tests\u002Funit_tests\u002Frecipes\u002Ftest_run_plugins.py`\n\n### Re-run state machine\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `RerunStateMachineConfig`\n- Init: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py` — `init_rerun_state`\n\n### Checkpointing\n- Async save: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `schedule_async_save`\n- Local ckpt: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `LocalCheckpointManager`\n- Tests: `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_local_checkpointing.py`\n\n## Pitfalls\n\n1. **ft_launcher, not torchrun**: Direct `FaultToleranceConfig` requires\n   `ft_launcher`. Using `torchrun` silently disables FT. For non-Slurm,\n   set `GROUP_RANK=0`.\n\n2. **Async save requires torch_dist**: `async_save=True` only works with\n   `ckpt_format=\"torch_dist\"`. Other formats silently fail or error.\n\n3. **IPR + NeMo-Run**: In-process restart is not compatible with NeMo-Run\n   or Slurm preemption plugins. Requires specific PyTorch\u002FNCCL versions\n   and env vars.\n\n4. **NVRx vs legacy straggler**: Two detectors exist. Use NVRx\n   (`nvrx_straggler`); do not enable both.\n\n5. **stop_if_detected default**: NVRx logs but does not stop training by\n   default. Set `stop_if_detected=True` for automatic termination.\n\n6. **NCCL watchdog vs hard_timeout**: For IPR, NCCL watchdog timeout must\n   exceed `hard_timeout` or PyTorch kills the process before recovery.\n\n7. **Rerun state machine is alpha**: Use `check_for_nan_in_loss=True` for\n   NaN detection, but don't rely on full rerun workflows yet.\n\n## Verification\n\n### Fault tolerance\n```bash\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh --simulate-fault\n```\nLook for `[FaultTolerance]` \u002F `[RankMonitorServer]` log lines with section\ntimeouts. Simulated fault should trigger restart from checkpoint.\n\n### Straggler detection\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n    examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002Fstraggler_detection_example.py\n```\nLook for `GPU relative performance` and `GPU individual performance` reports\nwith per-rank scores.\n\n### Async checkpoint\nLook for `Scheduling async checkpoint save` in logs. Training iterations\nshould continue while checkpoint files are being written.\n\n### In-process restart\n```bash\npytest tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py -v\n```\nRequires compatible PyTorch\u002FNCCL versions.\n","# Resiliency\n\nStable docs: @docs\u002Ftraining\u002Fresiliency.md, @docs\u002Ftraining\u002Fcheckpointing.md\nCard: @skills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml\n\n## Enablement\n\n### Fault tolerance (Slurm only)\n\n#### Option 1: NeMo Run plugin (recommended)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import FaultTolerancePlugin\nimport nemo_run as run\n\ntask = run.Script(...)\nrun_plugins = [\n    FaultTolerancePlugin(\n        enable_ft_package=True,\n        calc_ft_timeouts=True,\n        num_in_job_restarts=3,\n        num_job_retries_on_failure=2,\n        initial_rank_heartbeat_timeout=1800,\n        rank_heartbeat_timeout=300,\n    )\n]\nrun.run(task, plugins=run_plugins, executor=executor)\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `num_in_job_restarts` | 3 | Max restarts within same job |\n| `num_job_retries_on_failure` | 2 | Max new job launches on failure |\n| `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) |\n| `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) |\n\n#### Option 2: Direct config + ft_launcher\n\n```python\nfrom megatron.bridge.training.config import FaultToleranceConfig\n\ncfg.ft = FaultToleranceConfig(\n    enable_ft_package=True,\n    calc_ft_timeouts=True,\n    simulate_fault=False,\n    simulated_fault_type=\"random\",\n)\n```\n\nLaunch with `ft_launcher` (not `torchrun`):\n\n```bash\nexport GROUP_RANK=0  # required for non-Slurm\nft_launcher \\\n    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \\\n    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \\\n    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \\\n    --ft-rank_out_of_section_timeout=300 \\\n    your_training_script.py\n```\n\n| Config parameter | Default | Description |\n|---|---|---|\n| `enable_ft_package` | False | Enable fault tolerance |\n| `calc_ft_timeouts` | False | Auto-compute optimal timeouts |\n| `simulate_fault` | False | Enable fault simulation for testing |\n| `simulated_fault_type` | `\"random\"` | `\"rank_hung\"`, `\"rank_killed\"`, or `\"random\"` |\n| `simulated_fault_rank` | None | Specific rank to fault (random if None) |\n| `simulated_fault_base_delay` | 0 | Base delay before simulating fault |\n\nSection-based timeout monitoring covers setup, training steps, checkpointing,\nand out-of-section time independently. Timeouts are saved to `ft_state.json`\nfor subsequent runs when `calc_ft_timeouts=True`.\n\n### NVRx straggler detection\n\n```python\nfrom megatron.bridge.training.config import NVRxStragglerDetectionConfig\n\ncfg.nvrx_straggler = NVRxStragglerDetectionConfig(\n    enabled=True,\n    report_time_interval=300.0,\n    calc_relative_gpu_perf=True,\n    calc_individual_gpu_perf=True,\n    num_gpu_perf_scores_to_print=5,\n    gpu_relative_perf_threshold=0.7,\n    gpu_individual_perf_threshold=0.7,\n    stop_if_detected=False,\n    enable_logging=True,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable straggler detection |\n| `report_time_interval` | 300.0 | Seconds between straggler checks |\n| `calc_relative_gpu_perf` | True | Compare ranks against each other |\n| `calc_individual_gpu_perf` | True | Track per-rank degradation over time |\n| `gpu_relative_perf_threshold` | 0.7 | Threshold for relative performance (0-1) |\n| `gpu_individual_perf_threshold` | 0.7 | Threshold for individual performance (0-1) |\n| `stop_if_detected` | False | Terminate training on straggler |\n| `num_gpu_perf_scores_to_print` | 5 | Number of best\u002Fworst scores to print |\n| `profiling_interval` | 1 | Profiling interval for detector |\n\n### Preemption\n\n#### Plugin (Slurm)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import PreemptionPlugin\n\nplugins = [\n    PreemptionPlugin(\n        preempt_time=60,\n        enable_exit_handler=True,\n        enable_exit_handler_for_data_loader=False,\n    )\n]\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `preempt_time` | 60 | Seconds before job limit to send signal |\n| `enable_exit_handler` | True | Enable signal handler in training |\n| `enable_exit_handler_for_data_loader` | False | Enable for dataloader workers |\n\n#### Direct config\n\n```python\nimport signal\ncfg.train.exit_signal_handler = True\ncfg.train.exit_signal = signal.SIGTERM\ncfg.train.exit_signal_handler_for_dataloader = False\n```\n\n### Re-run state machine (experimental)\n\n```python\nfrom megatron.bridge.training.config import RerunStateMachineConfig\n\ncfg.rerun_state_machine = RerunStateMachineConfig(\n    rerun_mode=\"validate_results\",\n    check_for_nan_in_loss=True,\n    check_for_spiky_loss=False,\n    spiky_loss_factor=10.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `rerun_mode` | `\"disabled\"` | `\"disabled\"`, `\"validate_results\"`, `\"report_determinism_stats\"` |\n| `check_for_nan_in_loss` | True | Check for NaN in loss |\n| `check_for_spiky_loss` | False | Check for unexpectedly large loss |\n| `spiky_loss_factor` | 10.0 | Loss flagged if > factor * max observed (increase for large models) |\n\nExit codes: 16 = resume to disambiguate, 17 = failed validation.\n\n### In-process restart (experimental)\n\n```python\nfrom megatron.bridge.training.config import InProcessRestartConfig\n\ncfg.inprocess_restart = InProcessRestartConfig(\n    enabled=True,\n    granularity=\"node\",\n    soft_timeout=60.0,\n    hard_timeout=90.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable in-process restart |\n| `active_world_size` | None | Ranks executing workload (rest are warm reserves) |\n| `granularity` | `\"node\"` | `\"node\"` or `\"rank\"` restart granularity |\n| `max_iterations` | None | Max restart attempts (None = unlimited) |\n| `soft_timeout` | 60.0 | Detect GIL-released hangs (seconds) |\n| `hard_timeout` | 90.0 | Force-terminate hung ranks (seconds) |\n| `heartbeat_interval` | 30.0 | Heartbeat interval (seconds) |\n| `heartbeat_timeout` | 60.0 | Missing heartbeat timeout (seconds) |\n| `barrier_timeout` | 120.0 | Distributed barrier timeout (seconds) |\n| `completion_timeout` | 120.0 | Completion barrier timeout (seconds) |\n| `empty_cuda_cache` | True | Clear CUDA cache during restart |\n| `max_rank_faults` | None | Max rank faults before terminating |\n| `monitor_process_logdir` | None | Directory for monitor logs |\n\nRequired environment variables:\n\n```bash\nexport TORCH_CPP_LOG_LEVEL=error\nexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0\nexport NCCL_NVLS_ENABLE=0\n```\n\nThe PyTorch NCCL watchdog timeout must exceed `hard_timeout`. NeMo-Run's\nSlurm Executor is not supported; launch directly with `srun --kill-on-bad-exit=0`.\n\n### Async checkpoint save\n\n```python\ncfg.checkpoint.async_save = True\ncfg.checkpoint.ckpt_format = \"torch_dist\"\n```\n\n### Local checkpointing (NVRx)\n\n```python\ncfg.checkpoint.non_persistent_local_ckpt_dir = \"\u002Flocal\u002Fscratch\u002Fckpt\"\ncfg.checkpoint.non_persistent_local_ckpt_algo = \"fully_parallel\"\n```\n\n## Code Anchors\n\n### Fault tolerance\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `FaultToleranceConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ffault_tolerance.py`\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `FaultTolerancePlugin`\n- Perf plugin: `scripts\u002Fperformance\u002Fnemo-mbridge-resiliency_plugins.py`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_fault_tolerance.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002F`\n\n### Straggler detection\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `NVRxStragglerDetectionConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fnvrx_straggler.py`\n- Train loop: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py` — `check_nvrx_straggler_detection`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002F`\n\n### In-process restart\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `InProcessRestartConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finprocess_restart.py`\n- Entry point: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fpretrain.py` — `maybe_wrap_for_inprocess_restart`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_inprocess_restart.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py`\n\n### Preemption\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `PreemptionPlugin`\n- Signal handler: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Futils\u002Fsig_utils.py`\n- Tests: `tests\u002Funit_tests\u002Frecipes\u002Ftest_run_plugins.py`\n\n### Re-run state machine\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `RerunStateMachineConfig`\n- Init: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py` — `init_rerun_state`\n\n### Checkpointing\n- Async save: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `schedule_async_save`\n- Local ckpt: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `LocalCheckpointManager`\n- Tests: `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_local_checkpointing.py`\n\n## Pitfalls\n\n1. **ft_launcher, not torchrun**: Direct `FaultToleranceConfig` requires\n   `ft_launcher`. Using `torchrun` silently disables FT. For non-Slurm,\n   set `GROUP_RANK=0`.\n\n2. **Async save requires torch_dist**: `async_save=True` only works with\n   `ckpt_format=\"torch_dist\"`. Other formats silently fail or error.\n\n3. **IPR + NeMo-Run**: In-process restart is not compatible with NeMo-Run\n   or Slurm preemption plugins. Requires specific PyTorch\u002FNCCL versions\n   and env vars.\n\n4. **NVRx vs legacy straggler**: Two detectors exist. Use NVRx\n   (`nvrx_straggler`); do not enable both.\n\n5. **stop_if_detected default**: NVRx logs but does not stop training by\n   default. Set `stop_if_detected=True` for automatic termination.\n\n6. **NCCL watchdog vs hard_timeout**: For IPR, NCCL watchdog timeout must\n   exceed `hard_timeout` or PyTorch kills the process before recovery.\n\n7. **Rerun state machine is alpha**: Use `check_for_nan_in_loss=True` for\n   NaN detection, but don't rely on full rerun workflows yet.\n\n## Verification\n\n### Fault tolerance\n```bash\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh --simulate-fault\n```\nLook for `[FaultTolerance]` \u002F `[RankMonitorServer]` log lines with section\ntimeouts. Simulated fault should trigger restart from checkpoint.\n\n### Straggler detection\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n    examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002Fstraggler_detection_example.py\n```\nLook for `GPU relative performance` and `GPU individual performance` reports\nwith per-rank scores.\n\n### Async checkpoint\nLook for `Scheduling async checkpoint save` in logs. Training iterations\nshould continue while checkpoint files are being written.\n\n### In-process restart\n```bash\npytest tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py -v\n```\nRequires compatible PyTorch\u002FNCCL versions.\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Enabling resiliency features, or investigating a commit that caused training hangs, straggler detection failures, or broken restarts; 'fault tolerance', 'straggler detection', 'hang detection', 'automatic restart', 'in-process restart', 'preemption', 'nvidia-resiliency-ext'.","main","be3de0244dc1a57e323d8160787ebb40fb64930e","public\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main","https:\u002F\u002Fskills.x402.wtf","https:\u002F\u002Fgithub.com\u002FSolizardking","https:\u002F\u002Fmusebook.trade","Got a mile high and never came down","2026-10-08 23:20:13",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add Solizardking\u002Fskillhub-main --skill nemo-mbridge-resiliency",{"tool":40,"command":41},"bunx","bunx skills add Solizardking\u002Fskillhub-main --skill nemo-mbridge-resiliency",{"tool":43,"command":44},"pnpm","pnpm dlx skills add Solizardking\u002Fskillhub-main --skill nemo-mbridge-resiliency","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002FSKILL.md",true,[48,54,58,62,66],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"BENCHMARK.md","REFERENCE","参考文档",4267,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002FBENCHMARK.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"card.yaml",4910,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002Fcard.yaml",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"metadata.json",363,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002Fmetadata.json",{"path":63,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":64,"sourceHtmlUrl":65},"skill-card.md",3896,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002Fskill-card.md",{"path":67,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":68,"sourceHtmlUrl":69},"verification.json",1506,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fnemo-mbridge-resiliency\u002Fverification.json",[71,79,86,93,100,108],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":19},"01M78E0TGCAMNPS207BA3G07S7","zkrouter","Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.","MIT","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzkrouter\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M78E0R1Z8BDT6EDTHY17MTNB","zk","For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzk\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M78E0P3ZTZ1YWF8MD5MM23QE","zk-omni-messaging","Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzk-omni-messaging\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M78E0JXZ2PNB1896VXAGDBWD","zerion-wallet-data","Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzerion-wallet-data\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":104,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M78E0G14FH9GRRTQYMKD5E59","youtube-clipper","YouTube 视频智能剪辑工具。下载视频和字幕，AI 分析生成精细章节（几分钟级别）， 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频，并生成总结文案。 使用场景：当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词：视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M78E0C2ZYA99CFFDNB7ZMK5S","yield-optimization","Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit\u002Fwithdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter\u002Fexit a position, or check portfolio balances.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyield-optimization\u002FSKILL.md",[],[]]