Large language model applications — especially agentic systems that handle repeated or paraphrased user questions — often pay a heavy latency tax on every request. Most end-to-end delay comes not from prompt processing, but from autoregressive token generation: the model must produce output one token at a time, which can take seconds under load. Semantic caching offers a practical way to break that bottleneck at the application layer.

Unlike model-level KV caching, semantic caching works above the LLM. Incoming prompts are embedded into vectors and matched against previously cached requests using approximate nearest neighbor search. When a semantically similar question is found, the gateway returns a stored response directly — skipping expensive decode and, in many cases, downstream RAG or tool calls.

In this session, I share a production-oriented pattern built on Azure AI Gateway, Azure Managed Redis, and embedding models hosted on Microsoft Foundry. Using APIM policies such as llm-semantic-cache-lookup and llm-semantic-cache-store, the gateway implements a classic cache-aside flow: embed the prompt, search Redis for the closest match, route to the model on a miss, and store the result with a configurable TTL and similarity threshold. In a benchmark with semantically equivalent prompts, response time dropped from about 6 seconds on the first request to roughly 0.5 seconds on cache hits — a 12× speedup.

The session then goes deeper into the algorithm behind that lookup: Hierarchical Navigable Small World (HNSW), the same vector search technique widely used in RAG document retrieval. Starting from Navigable Small World graphs and skip lists, I explain how HNSW enables fast, scalable semantic matching — and how parameters such as score-threshold translate algorithm behavior into real cache hit/miss tradeoffs.

Attendees will leave with a clear mental model of semantic caching on Azure, reproducible architecture guidance, and practical tuning insights for agent workloads where latency and cost both matter.