Building AI agents is easier than ever with open source tools, but ensuring their reliability in production remains a major challenge. Unlike traditional software, AI agents are non-deterministic, making simple pass/fail testing insufficient.

This talk introduces a practical approach to evaluation and observability for AI agents, combining open source tools such as TruLens with agent architectures inspired by AgentGPT.

We will demonstrate how to instrument agent workflows, capture execution traces, and implement evaluation metrics such as faithfulness, tool selection accuracy, and answer relevancy. Attendees will also see how to visualize agent behavior and identify failure points across retrieval, reasoning, and generation layers using a lightweight dashboard.

Finally, we show how to build a feedback loop to iteratively improve agent performance, and share a reference implementation (GitHub) that can be reused with different agent frameworks.