Training and open sourcing frontier foundational models presents unique challenges—scaling to trillions of tokens across thousands of GPUs, maintaining reproducibility, and enabling a global community to fine-tune and deploy models. In this session, we explore how PyTorch addresses key challenges in building and open sourcing foundational LLMs (Muse Glimmer) and vision models (SAM).

We cover three angles: 1) how PyTorch-native tools like TorchTitan power pre-training through composable parallelism, torch.compile, and Float8—applied to both LLMs and vision architectures, 2) the road to open sourcing foundational models—from internal research to community-ready releases across modalities, and 3) how this foundation enables adopters in healthcare, robotics, and autonomous systems to fine-tune Muse Glimmer for domain NLP and SAM for zero-shot segmentation.

We share hard-won lessons: scaling distributed training without sacrificing code clarity, designing checkpointing for interoperability, and building an open ecosystem where models train on PyTorch and millions customize them. Attendees leave with practical strategies for training, releasing, and sustaining large-scale open models with PyTorch.