Unlike traditional software, where behaviour is deterministic and test outcomes are predictable, AI agents operate in dynamic, non-deterministic ways.

They can take different paths, make uncertain decisions, or even call the wrong tools, all while trying to complete a task.

In this talk, we’ll explore why standard benchmarks like MMLU or HellaSwag, designed for LLMs, fall short for agentic systems.

We’ll dive into "LLM as a Judge", Code based eval and human annotations techniques