# Self-Healing PyTorch Training Through Live GPU Migration

PyTorch Conference North America 2026 · [All sessions](https://sessionize.com/api/v2/s2rq62nk/view/markdown)

- **Session ID:** 1322707
- **When:** Wednesday, October 21, 2026 12:20 pm - 12:45 pm (Pacific Standard Time, UTC-08)
- **Room:** LL20C (Lower Level)
- **Speakers:** Pavneet Ahluwalia, Gavin Cohen

## Description

Anyone running large training jobs knows the 3am page: one link flap, GPU fault, or node failure, and the entire job is down. The cause is structural — every training step ends in a collective that needs all ranks. One dead or hung rank stalls the rest until the NCCL watchdog fires, and since a communicator cannot be rebuilt around a missing peer, the whole job aborts.
The standard recovery mechanism, checkpoint and restart, carries a heavy penalty: reprovisioning nodes, restarting the job, restoring the latest checkpoint, and recomputing every step since. A single failure typically burns 45–90 minutes of wall-clock time across every GPU in the job.
Together AI and Clockwork Systems will present the TorchPass powered training failover system, which keeps training jobs running through infrastructure failures. TorchPass reroutes traffic around degraded or failed links and, when a GPU, its memory, or another component fails, live-migrates the affected rank's device and communicator state to a spare resource — resuming from the same step with just a brief pause.
We'll walk through the architecture behind TorchPass, the end to end system from monitoring, health checks to failover and then demonstrate a live multi-node training job making continuous forward progress through injected link flaps and GPU failures. You'll leave with a clear model of when live migration beats checkpoint-restart, and a way to quantify recovered goodput and its dollar value for your own cluster using the ClusterMAX TCO Calculator.

\******************
_In order to facilitate networking and business relationships at the event, you may choose to visit a third party’s booth or access sponsored content. You are never required to visit third party booths or to access sponsored content. When visiting a booth or participating in sponsored activities, the third party will receive some of your registration data. This data includes your first name, last name, title, company, address, email, standard demographics questions (i.e. job function, industry), and details about the sponsored content or resources you interacted with. If you choose to interact with a booth or access sponsored content, you are explicitly consenting to receipt and use of such data by the third-party recipients, which will be subject to their own privacy policies._

## Speakers

- [Pavneet Ahluwalia](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/2e10b79f-062a-4a53-8634-92610cbb703a) — Product @ Microsoft Azure
- [Gavin Cohen](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/8ccbbde4-b35e-4835-80c6-7af8fcb74b49) — VP of Product at Clockwork Systems