Session
When a Tool Lies, Your Agent Believes It
Your tests assume tools behave. In production a tool times out, the network drops, or a call succeeds and hands back a corrupted result. That last one is the dangerous one: nothing throws an error, so the agent reports the bad value as fact.
Chaos testing injects those failures on purpose and scores the agent against ground truth. Then fix each failure type in code: check the result against the real record, fall back to a backup provider, and re-run everything to catch regressions.
What you'll learn:
• A chaos test you can add to an existing agent without changing its tools
• Ground-truth evaluators that catch the silent failure a status check misses
• A result-check hook and a fallback hook for both kinds of tool failure
• The habit of re-running the whole suite after every fix
• An open source notebook with the full loop
Outline:
• The Failure That Throws No Error
• Break the Tools on Purpose
• Fix the Silent One: Check the Result Against the Real Record
• Fix the Loud Ones: A Fallback Provider and an Honest Answer
• Re-run Everything: The Regression a Fix Introduced
• Build Chaos Into Your Test Suite
Elizabeth Fuentes Leone
Developer Advocate
San Francisco, California, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top