# Precompile for Training

PyTorch Conference North America 2026 · [All sessions](https://sessionize.com/api/v2/s2rq62nk/view/markdown)

- **Session ID:** 1245005
- **When:** Wednesday, October 21, 2026 12:00 pm - 12:10 pm (Pacific Standard Time, UTC-08)
- **Room:** LL20C (Lower Level)
- **Speakers:** Bob Ren, Aaron Orenstein

## Description

Distributed training systems today often compile the same model independently on every GPU rank. At small scale this is mostly an inconvenience. At large scale it becomes a major operational cost: thousands of GPUs can remain idle while identical compilation work is repeated across the cluster.

Beyond startup overhead, per-rank compilation introduces deeper systems challenges:

\* Different ranks can make slightly different compiler decisions even when executing the same SPMD program, leading to NCCL hangs and job failures.
\* Large training jobs waste substantial cluster capacity performing redundant compilation work across thousands of GPUs at runtime.
\* Independent autotuning can choose different kernels on different ranks, creating numerical inconsistencies and making debugging and reproducibility much harder.

This talk introduces Precompile for Training, a new abstraction for compiling distributed SPMD programs once and reusing the resulting artifact across all ranks. If every rank shares the same compiled graph artifact, compilation can move ahead of GPU allocation while guaranteeing identical execution and kernel selection across the cluster.

## Speakers

- [Bob Ren](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/fe5f994f-4af5-4ec0-b379-d17a581b641b) — Software Engineer
- [Aaron Orenstein](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/750d7a61-d753-4b28-b74d-dcb57d323772) — Software Engineer, Meta