Envoy was designed for traffic where requests are short-lived, predictable, and roughly equal in cost. AI inference breaks those assumptions. One request may stream for minutes, generate thousands of tokens, and consume far more compute than the one beside it.

I'll explain why these differences led to Envoy AI Gateway and how it extends Envoy Gateway for AI workloads. Using a single inference request, we'll see how AI-specific concepts like token-aware rate limiting, AIGatewayRoute, and AIServiceBackend fit into the familiar Envoy model.

Rather than covering every feature, this session focuses on building the mental model you'll need before configuring your first AI gateway. If you already understand Envoy but are new to serving LLMs, you'll leave knowing which networking instincts still apply, and which ones don't.