Session

From Tokens to Throughput: Operating vLLM at Production Scale

As LLMs move into production, inference has become a systems engineering problem rather than a modeling challenge. While vLLM enables high-throughput serving via PagedAttention and continuous batching, operating it reliably at scale exposes new bottlenecks. This talk explores real-world failure modes in production vLLM deployments, including tail-latency spikes, KV-cache fragmentation, and GPU under-utilization driven by memory bandwidth pressure. We walk through the inference request lifecycle, highlight where latency hides, and introduce vLLM-specific observability signals that traditional metrics miss. Attendees will gain a systems-level mental model and practical guidance for building reliable, high-performance LLM serving platforms with PyTorch.

Akkhil Gupta

DeduceTheLogic

Hyderābād, India

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top