Agent turns are mostly prefix: system prompts, tool schemas, prior turns, retrieved context. In a twenty-step loop, recomputing that prefix every time dominates latency.
For an 8B model with GQA, 32 layers, 8 KV heads, 128-dimensional heads, and FP16 KV cache, a 10,000-token prefix is about 1.2 GiB of KV state. vLLM reuses it within a replica. Then you autoscale behind Kubernetes, consecutive turns land on different pods, hit rates fall, dashboards stay green, and p99 TTFT quietly doubles.
This talk walks through what is happening under the hood: where KV blocks live across HBM, host DRAM, and a remote tier, how LMCache moves them through a RESP backend, and how Valkey acts as the shared cache. Plus the bandwidth math for a practical question: when is moving 1 GB of KV state faster than recomputing it on an H100?
Then the security issue we did not expect. Because blocks are content-addressed, two tenants with the same system prompt share cache entries by default. We show the isolation risk and how we closed it with Valkey database namespaces and database-level ACLs.
You leave with a working config, benchmark methodology, and the failure modes we hit first.