AI systems in production rarely fail due to obvious errors in model performance or system availability. Instead, they degrade through more subtle mechanisms: objective misalignment, feedback loop amplification, distribution shift induced by the system itself, and unintended interactions between model outputs and downstream business logic. In many cases, core metrics remain stable while real-world outcomes deteriorate, making these failures difficult to detect using traditional monitoring approaches.

This hands-on workshop introduces a structured, system-level approach to debugging production AI systems.

Drawing from large-scale search, recommendation, and advertising systems, the session frames AI products as multi-stage decision pipelines composed of retrieval, ranking, allocation, and feedback layers. Participants will learn how failures propagate across these layers and how to trace issues beyond model predictions into system behavior over time.

The workshop focuses on diagnosing four common classes of production failures:

1) Objective function misalignment between model optimization and business outcomes
2) Feedback loop dynamics that amplify bias or distort system behavior
3) System-induced distribution shift, where model outputs alter future training data
4) Local optimization effects that degrade global system performance

Participants will work through real-world scenarios to:

- Decompose AI systems into their decision layers and identify where control and logic reside
- Trace metric movement across stages of the pipeline to isolate root causes
- Analyze how user behavior and system outputs interact to create emergent failure modes
- Propose mitigation strategies, including constraint design, multi-objective optimization, and controlled exploration

By the end of the session, attendees will leave with:

1) A practical framework for debugging AI systems beyond model accuracy and offline metrics
2) Techniques for tracing failures across the end-to-end AI decision stack
3) Methods to detect and reason about feedback loops and system-driven distribution shift
4) Approaches to aligning model objectives with real-world system outcomes
5) A reusable mental model for designing more observable, robust, and production-ready AI systems