Most monitoring tells you what happened. Tracing tells you how it happened. Neither tells you whether your agent was right. For production AI agents, that last question is the only one that matters.
This talk builds the case for evaluation as the central engineering discipline of shipping AI agents and shows you exactly how to do it. We cover two foundational layers that work together:

OBSERVABILITY
Structured logging — what to capture at each agent step and how to structure it for queryability
Distributed tracing with ADOT — following a request through every tool call and model invocation, using AWS Distro for OpenTelemetry and CloudWatch

EVALUATION
Rule-based evals —deterministic checks that run on every response and catch regressions before users do
LLM-as-judge — a second model scoring your agent's outputs for correctness, relevance, and quality at scale
Human feedback collection — structured signals from real users that close the loop and drive continuous improvement

The result is a complete evaluation loop: observe, evaluate, improve, repeat.

You will leave with:
1. A mental model for evaluation-first AI agent design
2. A decision framework for choosing the right evaluation approach for each failure mode
3. An observability architecture built on ADOT and CloudWatch, designed specifically for AI agent workloads
4. Patterns for diagnosing where agents go wrong, separating prompt issues from tool failures from model limitations