Every demo serves one user. Production serves hundreds at once, and that's where things get ugly. One tenant fires off an 8K-token prompt and suddenly everyone else's chat completions are crawling, latency jumps 5x and your on-call gets paged at 2am.

We've been running multi-tenant inference using vLLM and llm-d, and honestly, the tricks that actually matter aren't the ones you'd expect. Disaggregated prefill and decode is the big one - splitting the compute-bound prompt crunching from the latency-sensitive token generation so they stop fighting each other. Prefix-cache-aware routing is the other, because if ten tenants share the same system prompt, you shouldn't process it ten times.

Beyond the single node, llm-d (now in CNCF Sandbox) lets you stretch vLLM across a Kubernetes cluster with KV offloading, cache-aware LoRA routing, and autoscaling that actually scales to zero when nobody's talking. We've seen ~3.1k tok/s per B200 GPU and TTFT improvements that are frankly hard to believe until you reproduce them.

In this talk, we’ll walk through the architecture we landed on, the mistakes we made getting there, and the observability techniques that saved us from shipping something embarrassing.

Audience:
Platform engineers, MLOps practitioners, and anyone shipping LLM-powered features to more than one customer.