TL;DR: A production story of scaling a multi-tenant AI gateway on Envoy to thousands of tenants — and the memory and config-size walls that stop you long before you expect.

Give every tenant their own models, limits, and providers and an Envoy-based AI gateway generates config per tenant, growing as tenants × models. Two walls hit early:
- The proxy's memory footprint (each upstream cluster costs heap)
- A low ceiling on how much routing config a gateway can hold, breached by only a handful of tenants.

This talk covers how we diagnosed both, the stopgap keeping production alive, and the fixes — sharing clusters, moving per-tenant policy onto the request, and changes we contributed upstream to Envoy Gateway — that scale to thousands of tenants without losing per-tenant behaviour.

Schedule:
- Intro — 2 min
- The scaling problem — 3 min
- Wall 1: memory footprint — 3 min
- Wall 2: config-size ceiling — 5 min
- Fixes + upstream work — 5 min
- Results & lessons — 3 min
- Q&A — 4 min