Session
Break Your Agent Before Production Does
Your agent passes every test you wrote, because every test assumes the world behaves. Production does not: tools return wrong data, and some users push the agent on purpose. Two kinds of trouble, and you need both. Bad luck is chaos. A weather tool returns 12 degrees for Miami in June, buried in junk, and the agent reports it as fact while paying tokens to read the noise. A guardrail that range-checks the value the moment a tool returns catches it, fixing accuracy and cost at once. Bad intent is red teaming. An attacker escalates over several turns until the agent leaks a stored card number or books above its limit. You generate these multi-turn attacks instead of scripting them, then score whether the agent held. Both are stochastic: run them once and you learn little, run them many times and you get the real picture, a correctness rate and a breach rate, not a single pass.
Outline: • The happy-path trap • Bad luck: chaos testing • Bad intent: red teaming • One run is not a measurement • Test both before you ship
Elizabeth Fuentes Leone
Developer Advocate
San Francisco, California, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top