# Scaling Foundation Models: From Broken to Near-Linear Multi-Node Training

PyTorch Conference North America 2026 · [All sessions](https://sessionize.com/api/v2/s2rq62nk/view/markdown)

- **Session ID:** 1256382
- **When:** Tuesday, October 20, 2026 2:50 pm - 3:00 pm (Pacific Standard Time, UTC-08)
- **Room:** LL21D (Lower Level)
- **Speakers:** Sheng Huang

## Description

At Pinterest, foundation models power recommendations for 600M+ MAUs. When we first tried multi-node training for a PyTorch/TorchRec model, adding a second 8-GPU node made training 5x slower than a single node. Even after Elastic Fabric Adapter (EFA) made networking viable, 4 nodes gave 1.21x scaling: 3x more GPUs for 21% more throughput. Larger model gains were out of reach.

We turned broken multi-node training into near-linear scaling. PyTorch Profiler and NCCL traces showed the bottleneck was not FLOPs, but cross-node All-to-All traffic from distributed embedding lookups: GPUs looked busy waiting on the network.

This talk covers: FP8 communication with FBGEMM, balanced TorchRec embedding sharding, embedding reshaping, Distributed Checkpoint (DCP) for load-time resharding, and a 2D parallel topology that keeps expensive All-to-All local and syncs replicas with AllReduce. Scaling improved from 1.13x to 2.0x at 2 nodes, and from 1.21x to 3.9x at 4 nodes, reaching 97.5% of ideal. It extended to 7.5x at 8 nodes, 93.75% of ideal.

Attendees leave with a playbook: measure bottlenecks, reduce bytes on the wire, checkpoint portably, and design topology around the slowest link.

## Speaker

- [Sheng Huang](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/de23d555-b39f-418a-bc40-59d20ae187d6) — Software Engineer, ML Training at Pinterest