Call add(2, 3) a thousand times and you get 5 every time. Ask an AI agent to process the same refund request three times and you may get three different answers, take three different paths through your tools, and still get an HTTP 200 on each one. Classic QA breaks: assertEquals fails on harmless variation, and one passing run proves almost nothing.

Physics hit the same wall a century ago and answered it with probability models instead of exact predictions. In this session we do the same for agents. We’ll look at where non-determinism really comes from (sampling, path, environment, supply chain), what an evaluation should measure (outcome and trajectory), and how to build an evaluation pyramid from cheap deterministic checks up to LLM-as-a-judge and human review. Then we’ll wire it into a release lifecycle with CI gates, shadow traffic, canaries and online evals, using observability on AWS as the measuring instrument. The examples come from a live refund agent built with Strands Agents and Amazon Bedrock.