When traditional software breaks, we have logs, traces, debuggers, and tools like git bisect. When an AI agent breaks, reproducing the failure can be much harder: the model may choose a different tool, take a different routing path, receive changed external data, or hand the task to a different agent.

This session presents a practical architecture for record-and-replay debugging of production AI agents. Every model invocation, context mutation, routing decision, MCP tool call, tool response, agent handoff, and side effect is captured as an execution graph rather than a flat log.

Using these traces, we will explore two debugging techniques. Execution diffing identifies the first meaningful divergence between successful and failed runs. Counterfactual replay keeps the original request and environment fixed while swapping one component, such as the model, prompt, routing policy, or tool, to determine whether a proposed change would actually have prevented the failure.

The session will walk through a multi-agent workflow that fails because of an incorrect routing decision, replay the original execution without repeating external side effects, test an alternative orchestration policy in shadow mode, and identify the decision responsible for the regression.

We will also cover the distributed-systems techniques that make this possible, including event sourcing, idempotency, trace propagation, immutable execution logs, side-effect isolation, and MCP-level instrumentation.

Attendees will leave with a concrete blueprint for answering four questions every production agent platform eventually faces: What failed? Where did it fail? Can I reproduce it? Would my fix actually work?