Remember back when we had to convince developers that unit tests, observability, and automated testing were important? We're now in the same situation with AI solutions.

With fast-moving LLMs and generative AI, "trust me" isn't enough to prove your solutions' functionality. AI outputs are inherently non-deterministic, and behavior can vary depending on the model, effort level, temperature settings, provider, and other factors. So, "it worked on my AI" isn't a sustainable approach. This is why we need AI-aware observability, not just traditional logging.

What defines a working AI solution? At a minimum, it needs to be measurable, monitorable, and reliable. We need evaluation frameworks that use automated evals, human-in-the-loop assessments, user feedback, and LLM-driven judging to verify expected behavior and detect regressions.

You wouldn't ship a traditional app without telemetry, so why treat AI differently? This talk covers the tools and practices developers need to build production-ready AI systems that are traceable, testable, monitored, and trustworthy.