AI agents are typically easy to demo but are more difficult to trust on production than traditional software. Not only can they return different results for the same prompt and context, a small prompt change, model switch, or tool response can change the behaviour of the whole application. So are there techniques that help us test a real system that has an AI agent(s) integrated in it?

This session does not replace the unit and integration tests you would already write for a JVM application. Instead, it focuses on the additional testing strategies that will help you test AI agents.

With a live demo, we will demonstrate a practical testing strategy. First, we will test the agent graph: verifying flow, decisions and tool calls without paying for model inference on every test run. This gives fast, deterministic feedback about the orchestration of the agent without incurring any model costs.

Then, we will use eval-driven development and "remocal" tests. We will update a prompt live, run evaluation tests, compare the results, and discuss how this approach helps ensure prompt and context quality, as well as help with switching/choosing models or preventing regressions.

Attendees will leave with a good understanding of how to test AI agents: graph tests for deterministic orchestration and evals for model-dependent behaviour and prompt quality.

I will need internet connection for the live demo