Anurag Aggarwal
Tetrate
Actions
Anurag is working in Networking domain for over 5 years. He has been a part of Flipkart's Networking team where he worked on service mesh and ebpf use cases for the first time in Indian Technical ecosystem. He carries the same experience to his current organisation "Tetrate" which is evolving the organisation to include newer projects and help kubernetes become affordable and scalable for everyone.
Links
Reverse Tunnels You Can Trust: Securing Remote Control in Envoy with JWT
TL;DR: A reverse tunnel lets a control plane reach into clusters it can't connect to — but only if you can trust who's dialing in. This talk shows how we authenticated Envoy reverse tunnels with JWT.
When your control plane runs in the cloud and data planes run in customers' clusters, traffic only flows outward — you can't reach in to reload config, revoke credentials, or debug live. Envoy's reverse tunnels invert that: the data plane dials out and the control plane sends requests back down. The catch is trust: the extension has no built-in auth. We authenticate every handshake with JWT (jwt_authn + remote_jwks), reusing identities the data plane already has.
Schedule:
- Intro — 2 min
- How reverse tunnels work — 2 min
- The trust problem — 2 min
- Authenticating with JWT — 2 min
- Q&A — 2 min
One Gateway, Thousands of Tenants: Scaling Envoy AI Gateway
TL;DR: A production story of scaling a multi-tenant AI gateway on Envoy to thousands of tenants — and the memory and config-size walls that stop you long before you expect.
Give every tenant their own models, limits, and providers and an Envoy-based AI gateway generates config per tenant, growing as tenants × models. Two walls hit early:
- The proxy's memory footprint (each upstream cluster costs heap)
- A low ceiling on how much routing config a gateway can hold, breached by only a handful of tenants.
This talk covers how we diagnosed both, the stopgap keeping production alive, and the fixes — sharing clusters, moving per-tenant policy onto the request, and changes we contributed upstream to Envoy Gateway — that scale to thousands of tenants without losing per-tenant behaviour.
Schedule:
- Intro — 2 min
- The scaling problem — 3 min
- Wall 1: memory footprint — 3 min
- Wall 2: config-size ceiling — 5 min
- Fixes + upstream work — 5 min
- Results & lessons — 3 min
- Q&A — 4 min
Replacing Traditional Hardware Load Balancers with eBPF and Maglev in Kubernetes
Traditional hardware load balancers introduce significant cost, complexity, and scalability challenges in Kubernetes environments. In this talk, we introduce an eBPF-based approach to Software-Defined Load Balancing (SD-LB) that eliminates the need for external hardware appliances while ensuring high availability and failover capabilities.
We will explore the role of eBPF in Kubernetes networking, focusing on its integration with kube-proxy and how externalTrafficPolicy—powered by an eBPF agent—can enable seamless failover, particularly in NodePort and anycast service scenarios.
Through a live demonstration and code walkthrough, we will showcase how a request sent to a Pod on one node can fail over to another node advertising the same IP—without requiring traditional hardware load balancers. We will also cover Direct Server Return (DSR) and discuss strategies to scale eBPF-based solutions for handling high ingress traffic.
AI-Native Routing with Envoy
TL;DR: A Kubernetes-native AI gateway that centralizes governance, cost control, and observability for LLM traffic — with usage-based rate limiting, model routing, and secure MCP integration built on Envoy.
The detailed version:
As Generative AI adoption accelerates, many teams are integrating Large Language Models (LLMs) into production systems. However, connecting applications directly to AI providers like OpenAI, Anthropic, or cloud-hosted models quickly introduces challenges around security, rate limiting, multi-provider routing, cost control, observability, and governance. Traditional API gateways are not designed with AI-specific traffic patterns in mind — especially token-based billing, usage quotas, or model-specific routing strategies.
This is where Envoy AI Gateway comes in.
Built on top of Envoy Gateway and powered by Envoy Proxy, Envoy AI Gateway is a Kubernetes-native, AI-aware gateway designed specifically to manage and secure LLM traffic. It extends the Kubernetes Gateway API model to introduce AI-centric capabilities such as usage-based rate limiting, model routing, policy enforcement, and integration with Model Context Protocol (MCP).
At a high level, Envoy AI Gateway acts as a unified control plane between your applications and multiple AI providers. Instead of hardcoding provider endpoints and credentials inside your services, applications call a single internal gateway endpoint. The gateway then handles provider authentication, model selection, traffic shaping, observability, and resilience policies. This abstraction allows platform teams to centralize governance while giving developers a simple and consistent API.
One of the most critical differences between AI traffic and traditional REST traffic is that AI workloads are usage-driven. Billing and throttling depend on token consumption rather than simple request counts. Envoy AI Gateway introduces usage-based rate limiting, enabling teams to define quotas based on token usage per user, tenant, or API key. This allows organizations to protect themselves from runaway costs while maintaining fair usage across teams.
Another important capability is support for the Model Context Protocol (MCP), which standardizes how AI models interact with external tools and contextual data sources. By integrating MCP at the gateway layer, Envoy AI Gateway enables secure and controlled tool access for LLM-powered applications — all without modifying application code.
Operationally, the gateway integrates seamlessly with Kubernetes. It leverages CRDs for configuration, follows GitOps-friendly workflows, and supports enterprise-grade observability through metrics and tracing. This makes it especially attractive for platform engineering teams building AI capabilities on Kubernetes clusters.
In this session, we will explore the architecture of Envoy AI Gateway, understand how it differs from traditional API gateways, and demonstrate how it can be used to securely operate AI workloads at scale. Attendees will walk away with a practical understanding of how to introduce governance, scalability, and cost control into their AI platform using cloud-native patterns.
Overall time schedule for this session.
Presenter intro - 2 mins
On AI gateway - 5 mins
Demo MCP - 5 mins
Demo Token rate limit - 5 mins
Demo Observability - 3 mins (if we can squeeze time, else only 2 demos)
Conclusion - 2 mins
Q&A - 3 mins
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top