Session
Continuous Evaluation & Monitoring for AI Agents
Agentic/GenAI applications aren’t fundamentally different from traditional software, but they fail differently: a prompt tweak, model upgrade or new tool can quietly degrade answers while every test stays green, because a wrong answer looks just like a right one. Continuous Evaluation (CE) scores every change for quality and safety before it ships, blocking regressions like a failing unit test. Continuous Monitoring (CM) watches quality in production and surfaces the failures no test anticipated. Together they form one loop: what monitoring finds becomes tomorrow’s PR and test.
This session walks that loop end to end with live demos. You’ll learn to design and write evaluators as code, from deterministic checks to LLM-as-judge, gate pull requests and deployments on thresholds and baseline regression checks that tolerate noisy judges, treat hard controls differently from averages, and use red teaming as adversarial evaluation. For monitoring: tracing, online evaluators on live traffic, metrics, dashboards and alerts as code, and turning production failures into new test cases and automated PR fixes. You’ll leave with a tool-agnostic blueprint and five practical rules.
Your pipeline can prove the code compiles and the tests pass, but not that your Agents still gives correct answers. This talk shows how to make answer quality both a build gate and a production signal. Continuous Evaluation blocks the regressions you can name. Continuous Monitoring finds the ones you can’t. The loop between them turns every production failure into a new test.
Many teams are now adding agentic/gen-ai features to systems they already ship with mature CI/CD, and they find that none of their existing gates can see quality. This session fills that gap with engineering practice, not tooling:
evaluators written and versioned like unit tests;
thresholds and baselines that cope with non-deterministic output;
hard controls for the checks that must never fail;
red teaming as adversarial testing;
monitoring built on traces, metrics and alerts as code.
Everything is shown live and end to end: a pull request, a blocked deploy, and a production failure that becomes the next test.
Two contrasting stacks, one assembled from Azure building blocks and one on LangSmith, make the point that the loop matters more than the tool. The core agent code and some evals happen to be in Python, but every pattern maps onto pipelines: evaluations run as tests in the same CI gates, and monitoring builds on OpenTelemetry, which almost every team already use. Attendees leave with a tool-agnostic blueprint, five practical rules and two public repositories to fork.
Repos used:
https://github.com/sohamda/azure-ai-redteam-eval
https://github.com/sohamda/kyb-agent
Soham Dasgupta
Deployed Engineer @ LangChain
Utrecht, The Netherlands
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top