# Five Bottlenecks, Five Fixes: Systematic Performance Diagnosis for PyTorch Distributed Training

PyTorch Conference North America 2026 · [All sessions](https://sessionize.com/api/v2/s2rq62nk/view/markdown)

- **Session ID:** 1254803
- **When:** Tuesday, October 20, 2026 4:55 pm - 5:20 pm (Pacific Standard Time, UTC-08)
- **Room:** 210A (Concourse Level)
- **Speakers:** Paulo Aragao, Aravind Neelakantan

## Description

Performance tuning distributed PyTorch training shouldn't be dark art. We present a systematic diagnostic methodology backed by five real case studies: (1) DataLoader delivering 1.2% GPU utilization — 83x gain by switching to memory-mapped indexed binary format; (2) unnecessary pipeline stages in torchtitan causing 28% throughput loss; (3) why 0.4% network utilization with MoE expert parallelism is a pipeline bubble, not a network problem; (4) communication-compute overlap analysis comparing FSDP2 vs pipeline parallelism producing completely different torch.profiler traces on identical hardware; (5) achieving 6.8x end-to-end improvement through structured profiling for a production autonomous driving model.

Each case study maps to our diagnostic decision framework: compute-bound, memory-bound, communication-bound, or I/O-bound — with specific torch.profiler signatures, NCCL debug patterns, and targeted PyTorch-level fixes. You leave with a repeatable checklist applicable to any distributed training workload.

## Speakers

- [Paulo Aragao](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/053d32d2-63dd-4815-a8dd-8136dcccff1c) — Principal, Applied AI Architect, AWS
- [Aravind Neelakantan](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/5ebe0f49-ab78-42b4-85c7-9def6615cae7) — Specialist Solutions Architect, Amazon Web Services