JingJing (Chris) Bao
Senior Software Engineer
Beijing, China
Actions
I am JingJing (Chris) Bao, a senior software engineer currently working at Schlumberger. Previously worked in the software division of ASML and Grab, with 10 years of cross-industry R&D experience spanning semiconductors, internet, and energy.
Specializes in AI/ML systems, scalable distributed architecture, real-time computing and visualization, and high-concurrency programming, with deep experience in integrating AI with software engineering and turning it into production-ready solutions.
Holds the Microsoft Azure AI certification as a technical expert and is also a Microsoft Certified Trainer, with long-term dedication to technical knowledge sharing and talent development
Area of Expertise
Topics
Semantic Cache on Azure: 12× Faster LLM Responses
Large language model applications — especially agentic systems that handle repeated or paraphrased user questions — often pay a heavy latency tax on every request. Most end-to-end delay comes not from prompt processing, but from autoregressive token generation: the model must produce output one token at a time, which can take seconds under load. Semantic caching offers a practical way to break that bottleneck at the application layer.
Unlike model-level KV caching, semantic caching works above the LLM. Incoming prompts are embedded into vectors and matched against previously cached requests using approximate nearest neighbor search. When a semantically similar question is found, the gateway returns a stored response directly — skipping expensive decode and, in many cases, downstream RAG or tool calls.
In this session, I share a production-oriented pattern built on Azure AI Gateway, Azure Managed Redis, and embedding models hosted on Microsoft Foundry. Using APIM policies such as llm-semantic-cache-lookup and llm-semantic-cache-store, the gateway implements a classic cache-aside flow: embed the prompt, search Redis for the closest match, route to the model on a miss, and store the result with a configurable TTL and similarity threshold. In a benchmark with semantically equivalent prompts, response time dropped from about 6 seconds on the first request to roughly 0.5 seconds on cache hits — a 12× speedup.
The session then goes deeper into the algorithm behind that lookup: Hierarchical Navigable Small World (HNSW), the same vector search technique widely used in RAG document retrieval. Starting from Navigable Small World graphs and skip lists, I explain how HNSW enables fast, scalable semantic matching — and how parameters such as score-threshold translate algorithm behavior into real cache hit/miss tradeoffs.
Attendees will leave with a clear mental model of semantic caching on Azure, reproducible architecture guidance, and practical tuning insights for agent workloads where latency and cost both matter.
AI Gateway as a Security Control Plane: Content, Access, and Operational Controls
As AI applications move from experimentation into production, security teams need more than model-level safeguards. They need a control plane that can enforce policy consistently across prompts, agents, tools, and model endpoints. This session presents the AI gateway as that control plane.
I’ll walk through a practical gateway-based security architecture built around three layers of control for production AI systems:
- Content controls inspect and block risky requests and responses, including prompt injection attempts, unsafe content, and sensitive data exposure.
- Access controls govern which users, apps, agents, and MCP tools can reach model and API endpoints through authentication and authorization policies.
- Operational controls provide the limits, monitoring, and enforcement mechanisms needed to keep AI workloads safe, reliable, and manageable at scale.
Using Azure API Management and Azure Content Safety as the implementation example, this session shows how these controls work together in a real deployment. Let’s discuss how to reduce AI risk, apply stronger enforcement at the gateway layer, and build safer AI applications for production use.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top