Session
Chaos Testing: When Your Agent Trusts Bad Data
A weather tool returns 12 degrees for Miami in June, buried in a verbose dump of logs. The real value is around 31, but your agent cannot tell it is wrong, so it reports 12 as fact and pays tokens to read the junk. Your tests never caught it, because every test assumes the tool behaves. That is chaos testing: feed the agent a wrong but believable value on purpose and see if it notices. It usually does not, because the value is plausible and nothing throws an error. The fix is a guardrail that runs the moment a tool returns, before the model sees it. It range-checks the value and, on failure, swaps the whole result for a one-line error. The bad value is gone and the token cost drops at once.
Outline: • The happy-path trap • Inject realistic corruption • The guardrail that catches it • One run is not a measurement • Build chaos into your test suite
Elizabeth Fuentes Leone
Developer Advocate
San Francisco, California, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top