AI can level up your software capabilities, but as any adventurer knows, powerful technologies introduce challenges akin to facing a rogue dragon: unexpected behavior and complex troubleshooting. In this talk, we'll explore how to harness AI/ML solutions while maintaining control. Using .NET Aspire and OpenTelemetry as our magic map, we'll navigate the labyrinth of the Microsoft.Extensions.AI, Agent Framework, and Qdrant-powered RAG to build a virtual game master for solo tabletop adventures that can improve over time using feedback, sentiment analysis, analytics, and automated agent response evaluation.

Along the way, we'll recount the journey of a few failed prototypes and see how a solid basis of evaluation and monitoring can help you succeed where many adventurers have failed in the tombs of annihilation. The topic is fun, but the challenges of optimizing AI systems are all too real. Thankfully, modern advances in evaluation, orchestration, and observability can stop your AI agents from going chaotic evil on you—or your users.