Nemo Mbridge Resiliency

nvidia/skills/skills/nemo-mbridge-resiliency

作者 nvidiacf5224d14250Apache-2.03.5K 个星标收录于 2026年10月8日更新于 2026年10月8日仓库今天更新

Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.

仅含说明DevOps & Cloud
AI 生成的概览

介绍 Megatron Bridge 的容错、掉队检测、抢占、重启与检查点等韧性功能。

功能
该技能是一份参考指南，用于启用和排查 Megatron Bridge 训练中的韧性功能。内容涵盖容错配置与启动器用法、NVRx 掉队检测、抢占信号处理、实验性重跑状态机、进程内重启，以及异步或本地检查点。它提供配置片段、参数表、代码位置、常见陷阱和验证命令。
适用场景
适用于在 Megatron Bridge 中启用韧性功能，或排查训练挂起、掉队检测失败、重启中断和抢占问题。也适合在确认某项韧性功能需要哪些配置参数和环境变量时使用。
运行要求
不附带脚本，仅为说明文档。按指南操作需要 Megatron Bridge 训练环境，包含 PyTorch、NCCL、NeMo-Run 和 nvidia-resiliency-ext 包；容错与抢占插件还需要 Slurm；进程内重启需要特定 PyTorch/NCCL 版本和环境变量。

Resiliency

Stable docs: @docs/training/resiliency.md, @docs/training/checkpointing.md Card: @skills/nemo-mbridge-resiliency/card.yaml

Enablement

Fault tolerance (Slurm only)

Option 1: NeMo Run plugin (recommended)
python
from megatron.bridge.recipes.run_plugins import FaultTolerancePluginimport nemo_run as run
task = run.Script(...)run_plugins = [    FaultTolerancePlugin(        enable_ft_package=True,        calc_ft_timeouts=True,        num_in_job_restarts=3,        num_job_retries_on_failure=2,        initial_rank_heartbeat_timeout=1800,        rank_heartbeat_timeout=300,    )]run.run(task, plugins=run_plugins, executor=executor)
Plugin parameterDefaultDescription
num_in_job_restarts3Max restarts within same job
num_job_retries_on_failure2Max new job launches on failure
initial_rank_heartbeat_timeout1800First heartbeat timeout (seconds)
rank_heartbeat_timeout300Subsequent heartbeat timeout (seconds)
Option 2: Direct config + ft_launcher
python
from megatron.bridge.training.config import FaultToleranceConfig
cfg.ft = FaultToleranceConfig(    enable_ft_package=True,    calc_ft_timeouts=True,    simulate_fault=False,    simulated_fault_type="random",)

Launch with ft_launcher (not torchrun):

bash
export GROUP_RANK=0  # required for non-Slurmft_launcher \    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \    --ft-rank_out_of_section_timeout=300 \    your_training_script.py
Config parameterDefaultDescription
enable_ft_packageFalseEnable fault tolerance
calc_ft_timeoutsFalseAuto-compute optimal timeouts
simulate_faultFalseEnable fault simulation for testing
simulated_fault_type"random""rank_hung", "rank_killed", or "random"
simulated_fault_rankNoneSpecific rank to fault (random if None)
simulated_fault_base_delay0Base delay before simulating fault

Section-based timeout monitoring covers setup, training steps, checkpointing, and out-of-section time independently. Timeouts are saved to ft_state.json for subsequent runs when calc_ft_timeouts=True.

NVRx straggler detection

python
from megatron.bridge.training.config import NVRxStragglerDetectionConfig
cfg.nvrx_straggler = NVRxStragglerDetectionConfig(    enabled=True,    report_time_interval=300.0,    calc_relative_gpu_perf=True,    calc_individual_gpu_perf=True,    num_gpu_perf_scores_to_print=5,    gpu_relative_perf_threshold=0.7,    gpu_individual_perf_threshold=0.7,    stop_if_detected=False,    enable_logging=True,)
ParameterDefaultDescription
enabledFalseEnable straggler detection
report_time_interval300.0Seconds between straggler checks
calc_relative_gpu_perfTrueCompare ranks against each other
calc_individual_gpu_perfTrueTrack per-rank degradation over time
gpu_relative_perf_threshold0.7Threshold for relative performance (0-1)
gpu_individual_perf_threshold0.7Threshold for individual performance (0-1)
stop_if_detectedFalseTerminate training on straggler
num_gpu_perf_scores_to_print5Number of best/worst scores to print
profiling_interval1Profiling interval for detector

Preemption

Plugin (Slurm)
python
from megatron.bridge.recipes.run_plugins import PreemptionPlugin
plugins = [    PreemptionPlugin(        preempt_time=60,        enable_exit_handler=True,        enable_exit_handler_for_data_loader=False,    )]
Plugin parameterDefaultDescription
preempt_time60Seconds before job limit to send signal
enable_exit_handlerTrueEnable signal handler in training
enable_exit_handler_for_data_loaderFalseEnable for dataloader workers
Direct config
python
import signalcfg.train.exit_signal_handler = Truecfg.train.exit_signal = signal.SIGTERMcfg.train.exit_signal_handler_for_dataloader = False

Re-run state machine (experimental)

python
from megatron.bridge.training.config import RerunStateMachineConfig
cfg.rerun_state_machine = RerunStateMachineConfig(    rerun_mode="validate_results",    check_for_nan_in_loss=True,    check_for_spiky_loss=False,    spiky_loss_factor=10.0,)
ParameterDefaultDescription
rerun_mode"disabled""disabled", "validate_results", "report_determinism_stats"
check_for_nan_in_lossTrueCheck for NaN in loss
check_for_spiky_lossFalseCheck for unexpectedly large loss
spiky_loss_factor10.0Loss flagged if > factor * max observed (increase for large models)

Exit codes: 16 = resume to disambiguate, 17 = failed validation.

In-process restart (experimental)

python
from megatron.bridge.training.config import InProcessRestartConfig
cfg.inprocess_restart = InProcessRestartConfig(    enabled=True,    granularity="node",    soft_timeout=60.0,    hard_timeout=90.0,)
ParameterDefaultDescription
enabledFalseEnable in-process restart
active_world_sizeNoneRanks executing workload (rest are warm reserves)
granularity"node""node" or "rank" restart granularity
max_iterationsNoneMax restart attempts (None = unlimited)
soft_timeout60.0Detect GIL-released hangs (seconds)
hard_timeout90.0Force-terminate hung ranks (seconds)
heartbeat_interval30.0Heartbeat interval (seconds)
heartbeat_timeout60.0Missing heartbeat timeout (seconds)
barrier_timeout120.0Distributed barrier timeout (seconds)
completion_timeout120.0Completion barrier timeout (seconds)
empty_cuda_cacheTrueClear CUDA cache during restart
max_rank_faultsNoneMax rank faults before terminating
monitor_process_logdirNoneDirectory for monitor logs

Required environment variables:

bash
export TORCH_CPP_LOG_LEVEL=errorexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0export NCCL_NVLS_ENABLE=0

The PyTorch NCCL watchdog timeout must exceed hard_timeout. NeMo-Run's Slurm Executor is not supported; launch directly with srun --kill-on-bad-exit=0.

Async checkpoint save

python
cfg.checkpoint.async_save = Truecfg.checkpoint.ckpt_format = "torch_dist"

Local checkpointing (NVRx)

python
cfg.checkpoint.non_persistent_local_ckpt_dir = "/local/scratch/ckpt"cfg.checkpoint.non_persistent_local_ckpt_algo = "fully_parallel"

Code Anchors

Fault tolerance

  • Config: src/megatron/bridge/training/config.py — FaultToleranceConfig
  • Runtime: src/megatron/bridge/training/fault_tolerance.py
  • Plugin: src/megatron/bridge/recipes/run_plugins.py — FaultTolerancePlugin
  • Perf plugin: scripts/performance/nemo-mbridge-resiliency_plugins.py
  • Tests: tests/unit_tests/training/test_fault_tolerance.py
  • Example: examples/training_features/nemo-mbridge-resiliency/fault_tolerance/

Straggler detection

  • Config: src/megatron/bridge/training/config.py — NVRxStragglerDetectionConfig
  • Runtime: src/megatron/bridge/training/nvrx_straggler.py
  • Train loop: src/megatron/bridge/training/train.py — check_nvrx_straggler_detection
  • Tests: tests/unit_tests/training/test_nvrx_straggler.py, tests/functional_tests/training/test_nvrx_straggler.py
  • Example: examples/training_features/nemo-mbridge-resiliency/straggler_detection/

In-process restart

  • Config: src/megatron/bridge/training/config.py — InProcessRestartConfig
  • Runtime: src/megatron/bridge/training/inprocess_restart.py
  • Entry point: src/megatron/bridge/training/pretrain.py — maybe_wrap_for_inprocess_restart
  • Tests: tests/unit_tests/training/test_inprocess_restart.py, tests/functional_tests/training/test_inprocess_restart.py

Preemption

  • Plugin: src/megatron/bridge/recipes/run_plugins.py — PreemptionPlugin
  • Signal handler: src/megatron/bridge/training/utils/sig_utils.py
  • Tests: tests/unit_tests/recipes/test_run_plugins.py

Re-run state machine

  • Config: src/megatron/bridge/training/config.py — RerunStateMachineConfig
  • Init: src/megatron/bridge/training/initialize.py — init_rerun_state

Checkpointing

  • Async save: src/megatron/bridge/training/checkpointing.py — schedule_async_save
  • Local ckpt: src/megatron/bridge/training/checkpointing.py — LocalCheckpointManager
  • Tests: tests/functional_tests/training/test_local_checkpointing.py

Pitfalls

  1. ft_launcher, not torchrun: Direct FaultToleranceConfig requires ft_launcher. Using torchrun silently disables FT. For non-Slurm, set GROUP_RANK=0.

  2. Async save requires torch_dist: async_save=True only works with ckpt_format="torch_dist". Other formats silently fail or error.

  3. IPR + NeMo-Run: In-process restart is not compatible with NeMo-Run or Slurm preemption plugins. Requires specific PyTorch/NCCL versions and env vars.

  4. NVRx vs legacy straggler: Two detectors exist. Use NVRx (nvrx_straggler); do not enable both.

  5. stop_if_detected default: NVRx logs but does not stop training by default. Set stop_if_detected=True for automatic termination.

  6. NCCL watchdog vs hard_timeout: For IPR, NCCL watchdog timeout must exceed hard_timeout or PyTorch kills the process before recovery.

  7. Rerun state machine is alpha: Use check_for_nan_in_loss=True for NaN detection, but don't rely on full rerun workflows yet.

Verification

Fault tolerance

bash
./examples/training_features/nemo-mbridge-resiliency/fault_tolerance/run_fault_tolerance.sh./examples/training_features/nemo-mbridge-resiliency/fault_tolerance/run_fault_tolerance.sh --simulate-fault

Look for [FaultTolerance] / [RankMonitorServer] log lines with section timeouts. Simulated fault should trigger restart from checkpoint.

Straggler detection

bash
uv run python -m torch.distributed.run --nproc_per_node=2 \    examples/training_features/nemo-mbridge-resiliency/straggler_detection/straggler_detection_example.py

Look for GPU relative performance and GPU individual performance reports with per-rank scores.

Async checkpoint

Look for Scheduling async checkpoint save in logs. Training iterations should continue while checkpoint files are being written.

In-process restart

bash
pytest tests/functional_tests/training/test_inprocess_restart.py -v

Requires compatible PyTorch/NCCL versions.

来源与署名

来源：nvidia/skills位于skills/nemo-mbridge-resiliency提交cf5224d

许可证： Apache-2.0

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 nvidia/skills 的技能

Tilegym Cutile Python

nvidia

指导编写、调试和优化 cuTile Python GPU 内核，包含示例、验证循环以及可选的多智能体编排。

包含脚本
Software Development3.5K今天更新

Tilegym Monkey Patch Kernels To Transformers

nvidia

通过非侵入式 monkey-patch 将 TileGym cuTile 内核集成到 Hugging Face transformers 模型，并自动生成新内核。

Software Development3.5K今天更新

Tilegym Converting Cutile To Julia

nvidia

将 cuTile Python GPU 内核转换为 cuTile.jl Julia 等价实现，并提供规则、示例与静态校验器。

包含脚本
Software Development3.5K今天更新

Tilegym Adding Cutile Kernel

nvidia

指导向 TileGym 添加新的 cuTile GPU 内核算子，涵盖调度注册、测试与基准测试。

Software Development3.5K今天更新

Physicsnemo Discover

nvidia

为 SciML 任务指引查找 PhysicsNeMo 的模型族、数据管道与示例，并引用实时仓库路径。

AI & Agents3.5K今天更新

Nemoclaw User Guide

nvidia

引导 AI 代理访问 NemoClaw 官方 Markdown 文档与文档 MCP 服务器，回答安装与运维问题。

Learning & Education3.5K今天更新

更多DevOps & Cloud技能

Playwright Devops

microsoft

精选

面向 Playwright 的 DevOps 工作流：分析 main 分支最新提交的 GitHub Actions 失败并下载失败作业日志。

包含脚本
DevOps & Cloud97K今天更新

M5 Onboard

anthropics

精选

通过 USB 检测 M5Stack ESP32 开发板，刷写 UIFlow 2.0 固件并安装 MicroPython 应用包。

DevOps & Cloud37K今天更新

Yeet

openai

精选

使用 GitHub CLI 一次性完成暂存、提交、推送并创建或更新 GitHub 拉取请求。

DevOps & Cloud27K4周前更新

Runbook

anthropics

精选

为重复性任务创建或更新分步运维手册，包含故障排查、回滚和升级流程。

DevOps & Cloud27K今天更新

Incident Response

anthropics

精选

指导事故响应流程：严重级别判定、状态更新、缓解跟踪以及无责事后复盘。

DevOps & Cloud27K今天更新

Deploy Checklist

anthropics

精选

生成部署前就绪检查清单，涵盖部署前、部署、部署后和回滚触发条件。

DevOps & Cloud27K今天更新