TL;DR: A Kubernetes-native AI gateway that centralizes governance, cost control, and observability for LLM traffic — with usage-based rate limiting, model routing, and secure MCP integration built on Envoy.

The detailed version:

As Generative AI adoption accelerates, many teams are integrating Large Language Models (LLMs) into production systems. However, connecting applications directly to AI providers like OpenAI, Anthropic, or cloud-hosted models quickly introduces challenges around security, rate limiting, multi-provider routing, cost control, observability, and governance. Traditional API gateways are not designed with AI-specific traffic patterns in mind — especially token-based billing, usage quotas, or model-specific routing strategies.

This is where Envoy AI Gateway comes in.
Built on top of Envoy Gateway and powered by Envoy Proxy, Envoy AI Gateway is a Kubernetes-native, AI-aware gateway designed specifically to manage and secure LLM traffic. It extends the Kubernetes Gateway API model to introduce AI-centric capabilities such as usage-based rate limiting, model routing, policy enforcement, and integration with Model Context Protocol (MCP).

At a high level, Envoy AI Gateway acts as a unified control plane between your applications and multiple AI providers. Instead of hardcoding provider endpoints and credentials inside your services, applications call a single internal gateway endpoint. The gateway then handles provider authentication, model selection, traffic shaping, observability, and resilience policies. This abstraction allows platform teams to centralize governance while giving developers a simple and consistent API.

One of the most critical differences between AI traffic and traditional REST traffic is that AI workloads are usage-driven. Billing and throttling depend on token consumption rather than simple request counts. Envoy AI Gateway introduces usage-based rate limiting, enabling teams to define quotas based on token usage per user, tenant, or API key. This allows organizations to protect themselves from runaway costs while maintaining fair usage across teams.

Another important capability is support for the Model Context Protocol (MCP), which standardizes how AI models interact with external tools and contextual data sources. By integrating MCP at the gateway layer, Envoy AI Gateway enables secure and controlled tool access for LLM-powered applications — all without modifying application code.

Operationally, the gateway integrates seamlessly with Kubernetes. It leverages CRDs for configuration, follows GitOps-friendly workflows, and supports enterprise-grade observability through metrics and tracing. This makes it especially attractive for platform engineering teams building AI capabilities on Kubernetes clusters.

In this session, we will explore the architecture of Envoy AI Gateway, understand how it differs from traditional API gateways, and demonstrate how it can be used to securely operate AI workloads at scale. Attendees will walk away with a practical understanding of how to introduce governance, scalability, and cost control into their AI platform using cloud-native patterns.

Overall time schedule for this session.

Presenter intro - 2 mins
On AI gateway - 5 mins
Demo MCP - 5 mins
Demo Token rate limit - 5 mins
Demo Observability - 3 mins (if we can squeeze time, else only 2 demos)
Conclusion - 2 mins
Q&A - 3 mins