# EP-Overlap: Hiding Expert-Parallel Communication in MoE Training

PyTorch Conference North America 2026 · [All sessions](https://sessionize.com/api/v2/s2rq62nk/view/markdown)

- **Session ID:** 1257875
- **When:** Tuesday, October 20, 2026 3:25 pm - 3:35 pm (Pacific Standard Time, UTC-08)
- **Room:** LL21D (Lower Level)
- **Speakers:** Sanket Jayant Purandare, Syed Ahmed

## Description

MoE training stalls 35-60% of every step on expert-parallel all-to-all comms, and MoE's dynamically sized activations force worst-case activation memory at every layer: O(worst_case x L).

EP-Overlap hides EP comm latency by splitting each MoE layer into independent slices and overlapping one slice's token exchange with any available compute (experts, activation recompute, weight-gradient math), forward-with-forward and backward-with-backward. Under PP it also overlaps forward with backward across microbatches, where no splitting is needed. It works under FSDP and PP across all PP phases with any schedule, not just DualPipe, running as graph passes in TorchTitan's GraphTrainer, composing with GraphPP, needing no model changes, and matching eager numerics bitwise.

The same machinery hides a second cost: page stashing, which compacts a layer's worst-case activations into a page stash, cutting activation memory to O(worst_case + actual bytes used). Its stash traffic is hidden the same way, making the savings near-free.

On DeepSeek-V3 we hide 76-86% of EP comm, giving ~1.3x throughput and +4% absolute MFU, a lower bound that sync-free routing (Hybrid-EP) and CUDA Graphs should push higher.

## Speakers

- [Sanket Jayant Purandare](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/08dcdb2a-60ca-496f-9e0e-7fb3efd6868c) — Research Engineer, PyTorch Distributed and Compilers
- [Syed Ahmed](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/a7e5e154-5275-4a0e-a7e9-58e202dfc364) — Senior Software Engineer, PyTorch Core @NVIDIA