NVIDIA open-sources and releases the NVIDIA Recovery workload recovery service
On November 3, 2026, NVIDIA released the open-source workload-level recovery service NVIDIA Recovery. The service detects training interruptions through application logs, GPU metrics, and scheduler signals, and processes them in a four-phase loop: detection, attribution, isolation, and restart or stop. The service is released under the Apache 2.0 license in the dsx-ai-factory/NVIDIA-Recovery repository; Slurm integration is implemented through an Advisor Controller deployed via a Helm chart, supporting both in-cluster and SSH modes. New deployments default to monitoring mode, which only outputs recovery recommendations and does not execute scheduling actions. From July 9 to October 9, monitoring on a training cluster covered nearly 200 jobs per day, recording approximately 8,550 anomaly detections, with average detection and recovery action latencies of 69 seconds and 94 seconds, respectively. The service is currently released as open source, and there is no information on further developments yet.
On November 3, 2026, NVIDIA released the open-source workload-level recovery service NVIDIA Recovery. The service detects training interruptions through application logs, GPU metrics, and scheduler signals, and processes them in a four-phase loop: detection, attribution, isolation, and restart or stop. The service is released under the Apache 2.0 license in the dsx-ai-factory/NVIDIA-Recovery repository; Slurm integration is implemented through an Advisor Controller deployed via a Helm chart, sup
Matched fieldsTitle