# Beyond Collectives: Building Fused Compute-Communication Kernels with PyTorch Symmetric Memory

PyTorch Conference North America 2026 · [All sessions](https://sessionize.com/api/v2/s2rq62nk/view/markdown)

- **Session ID:** 1257030
- **When:** Wednesday, October 21, 2026 11:45 am - 12:10 pm (Pacific Standard Time, UTC-08)
- **Room:** 210B (Concourse Level)
- **Speakers:** Ke Wen, Natalia Gimelshein, Kapil Sharma

## Description

Collective ops gave PyTorch portable distributed training, but they draw a hard line between compute and communication: a collective is a black box compute waits on. PyTorch Symmetric Memory erases that line, giving every rank a direct window into its peers' GPU memory so communication becomes something a kernel does inline. A practical tour across five themes:

\- Compute–communication fusion: pushing partial results to a peer from inside a kernel, hiding communication behind compute.
\- Flexible patterns: one-sided puts/gets and torch.distributed.TokenSwitch, supporting expert-parallel MoE with autograd for free.
\- DSL support: binding low-level NCCL device primitives into a CUDA DSL (cute-DSL).
\- Profiling & CUDA Graph: new device-side APIs that make fused communication observable, plus CUDA Graph capture for production loops.
\- Community adoption: where Symmetric Memory is landing today.

Attendees leave understanding the programming model and when fusing communication beats a collective. Assumes PyTorch distributed basics; deep CUDA not required.

## Speakers

- [Ke Wen](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/f9b10346-995d-4c20-a3a4-fdabef35fb13) — NVIDIA, Principal Software Architect, PyTorch Maintainer
- [Natalia Gimelshein](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/cf9b7b36-8fcb-4506-9215-566c6bc442d0) — Meta
- [Kapil Sharma](https://sessionize.com/api/v2/s2rq62nk/view/markdown/speaker/77fd5103-92a9-4c3e-a919-726a80e8f4bd) — Kapil Sharma, Meta, PyTorch Distributed