Ask an agent to read a 200-page contract and then hold a conversation about it, and watch what happens to your GPU memory. Every token of context the model has ever seen sits in a KV cache, and that cache grows with every turn, every tool call, every document you feed it. GPU memory doesn't grow to match. Eventually you hit the context wall: either you buy more
GPUs you don't need for compute, or you start truncating context.

We've been running long-context and agentic workloads, and the answer is being smarter about where the memory lives. A lot of an agentic workload is repeated prefix, and once you see that, you can budget context very differently. Here's what makes the difference:

- Tiered KV cache offloading keeps recent context fast and lets cold context get cheap.
- Prefill and decode disaggregation splits the compute-bound job of ingesting a huge prompt from the memory-bandwidth-bound job of generating tokens one at a time.
- KV-cache-aware routing sends a request to whichever replica already holds the relevant cache instead of recomputing it cold.
- Distributed autoscaling that watches KV-cache saturation and queue depth instead of GPU utilization, which can be a red herring under continuous batching.

We'll go through the architecture, how to move the bottleneck off memory bandwidth, and the monitoring signals that tell you you're approaching the wall before your agent starts silently forgetting things.

Audience:
Platform engineers building agentic systems, decision makers at companies with hardware available