Does your AI assistant tells its users to “add glue to their pasta recipe”, as well?
Installing an OpenAI client library and adding a few LLM calls to your code only take five minutes; but often enough, the real work only begins when you take your app into production and it starts hallucinating wildly. In this case, you’ll want:

1. insights into how users actually use your system
2. visibility into where things went wrong
3. confidence that your app produces outputs in the desired quality

This talk covers the essentials of running LLM-powered applications in production, focusing on observability and evaluations. We'll explore how known standards like OpenTelemetry can be adapted to trace complex LLM processes, surface meaningful metrics for monitoring, and continuously assess output quality. Along the way, we'll look at open-source tools that help you gain visibility into your AI systems and notice early when your app starts confidently telling users to add glue to their pasta.