Bringing a new accelerator up as a first-class PyTorch backend at scale takes more than vendor patches. It requires reproducible benchmarking, portable training code, and operational machinery that survives long-running jobs. This lightning talk shares experiences from training scientific foundation models in production on 63K+ XPUs on ALCF's Aurora system.

We cover three things:

- What was needed: the XPU integration path (torch.xpu, the IPEX-to-xccl transition, distributed primitives, collective tuning) and the cross-vendor abstractions that let identical training code run on Intel, NVIDIA, and AMD hardware.

- What was accomplished: migrating our production training stack from a Megatron-DeepSpeed fork to torchtitan + DTensor + FSDP2, with optimization and testing at every layer; from fused kernels and collectives up through distributed orchestration.

- What remains: fused kernels, async checkpointing, fault tolerance, and the operational layer long jobs require (silent-hang detection, bad-node failover, layered restart loops), plus what the next accelerator team should expect.

The takeaway: Patterns for anyone running PyTorch at scale on a new accelerator.

https://samf.sh/talks/2026/10/21/

https://samf.sh/talks/2026/10/21/