Let's face it - your AI app is probably broken. Maybe it calls the wrong tools, retrieves the wrong data from a RAG system, or even hallucinates more than a 1970's music festival audience.

The fun of developing AI apps is their non-deterministic nature. You can't write unit tests the way you are used to, so TDD is out. Even QA may pass your app only for it to fail in prod with different inputs, or model updates.

In this session, I propose a new methodology to shift left with AI app development, "evaluation driven development". Evaluations allow you to use AI and other tools to score your AI apps capabilities and responses, using more powerful or specialized models to evaluate the ones you use day to day.

If you build in evaluations during development, you can have more confidence in what you are shipping to QA, staging, or production. Evaluation driven development even allows you to bring in AI governance earlier in your development cycles, using evaluations from your compliance teams as part of your development processes.

By the end of this session you will understand how this new methodology can ensure quality and compliance in the apps you build.