NVIDIA Open-Sources NVIDIA Recovery to Automatically Recover Distributed Training Jobs on Clusters
NVIDIA has released NVIDIA Recovery, an open-source workload-level recovery service that detects training interruptions through application logs, GPU metrics, and scheduler signals, and handles them in a four-stage loop: detection, attribution, isolation, and restart or stop. The service is available under the Apache 2.0 license in the dsx-ai-factory/NVIDIA-Recovery repository. Slurm integration is implemented through an Advisor Controller deployed via a Helm chart, supporting both in-cluster and SSH modes. New deployments default to monitoring mode, which only outputs recovery recommendations without executing scheduling actions; from July 9 to October 9, monitoring on one training cluster covered nearly 200 jobs per day and recorded approximately 8,550 anomaly detections, with average detection and recovery action latencies of 69 seconds and 94 seconds, respectively.

NVIDIA has released NVIDIA Recovery, an open-source workload-level recovery service that detects training interruptions through application logs, GPU metrics, and scheduler signals, and handles them in a four-stage loop: detection, attribution, isolation, and restart or stop. The service is available under the Apache 2.0 license in the dsx-ai-factory/NVIDIA-Recovery repository. Slurm integration is implemented through an Advisor Controller deployed via a Helm chart, supporting both in-cluster an
Matched fieldsTitle、Summary、Body




