In today's world of fast moving LLMs and generative AI, "trust me" isn't a strategy—it's a liability. AI outputs are inherently non‑deterministic, especially given the factors of different models, temperature settings, provider switches and more. So, “it worked on my AI” is not a sustainable approach. This is why you need AI-aware observability, not just classic logging.

What really defines a working AI solution? At least, it has to be measurable, monitorable, and reliable. We'll cut through some important tracking metrics, including latency, throughput, error rates and token usage.

Since AI is probabilistic, not deterministic like traditional software, we need robust evaluation frameworks. We'll need things like automated evals, human-in-the-loop assessment, user-feedback, or even LLM-driven judging, to confirm expected behavior and detect regressions.

You wouldn't ship a classic app without telemetry, so why treat AI differently? This talk equips developers with the mindset and tools to build AI systems that are intelligent and production-ready: traceable, testable, monitored, and trustworthy.