Artificial intelligence and machine learning can significantly level up your software capabilities, offering exciting possibilities for applications like interactive storytelling and game mastering in Dungeons and Dragons. However, as any adventurer knows, these powerful technologies can also introduce challenges akin to facing a rogue dragon—unexpected behavior and complex troubleshooting. In this talk, we’ll explore how to harness the benefits of AI/ML solutions while maintaining control and minimizing unpredictability. We’ll use observability through OpenTelemetry as our magic map to gain insights into the labyrinth of AI orchestration systems, whether through frameworks like Semantic Kernel or custom telemetry collection. Additionally, we will examine how features like aggregate and individual feature importance in machine learning can help demystify and manage these AI systems, much like identifying the strengths and weaknesses of a mythical creature.

Furthermore, this session will cover the application of automated testing to AI and machine learning systems—a bit like ensuring your thief can disarm traps before the party walks into an ambush. We'll discuss the implementation of unit and integration tests for both LLM-based systems and traditional machine learning approaches. The role of data analytics will also be highlighted, emphasizing data-driven decision-making by analyzing usage patterns, processing time, confidence scores, user behavior, and more. Finally, we'll share strategies for taming unruly AI/ML systems in controlled and safe ways, illustrated with engaging examples from a Dungeons and Dragons AI system alongside anonymized real-world cases. By the end of this comprehensive session, you'll be equipped with the knowledge to control your AI solutions effectively and tackle AI-related challenges with the confidence of a seasoned adventurer, all while having a bit of fun along the way.