Your agent test suite is green. For each of those passes, ask a harder question. Who decided it was a pass, you or the agent?

Most agent testing takes the agent's word for what it did. The agent replies "I refused." The suite records a pass. Everyone goes home. An agent reporting its own outcome is an agent grading its own homework, and it will always pass.

Having come from quality background, we need to design experiment for failure. Unlike assertion, design of experiment is a real Test of the product.

Crucible is an open-source fitness gate built on one rule. The machine judges the agent. The agent never judges itself.

Point it at your agent and one command springs a floor of deterministic villains. No LLM on the trap side, so runs are free and reproduce byte for byte from a seed. Every verdict comes back as one of three things, and the third one is the argument.

SPRUNG. The agent performed the failure the trap was set for. A real observed action, replayable from the seed. A green never hides a sprung trap.

HELD. The agent defended, and the report discloses what the defence rests on. A real tool call Crucible mapped and judged, or a bare {"effect": "refuse"} the agent picked for itself. Every run prints the tally: N of M holds from mapped tool calls, K from self-named effects. Switch on --require-tool-verdict and a self-named green fails your build.

NOT_ARMED. Honestly untested. A trap you never set, or a prose reply carrying no observable action. Crucible never invents a verdict from prose.

Every dashboard reports what it tested. Almost none reports what it failed to test. Agent risk lives in the second number.

The eight tripwires are frozen: irreversible action, instruction lost after compaction, hostile service capture, silent stall, money leak, secret disclosure, authority escalation, forged identity acceptance. Four cover how an agent mismanages its own context and actions. Four cover money, secrets, authority and identity. The eight stay frozen so verdicts stay comparable. The scenario suites grow, and there are 14 domains and 56 real hostile situations today.

Then the stochastic problem, which is the one you have. Your agent refuses the trap when you try it by hand and leaks on the tenth customer, and you never reproduce the ghost. Crucible runs N trials, fails closed, and reports a rate with a 95% credible interval and the number of trials to tighten it. Sprang in 13 of 20, 43 to 82 percent. A rate, an interval, and the N it cost. Never a score.

Last, the case for this room. Judging an agent you did not build, over the wire, and handing a reproducible evidence bundle to a risk committee. You are unable to unit-test an agent you do not own. You probe it from outside and demand proof.

Apache-2.0. Local, free, LLM-free, no account. Clone it during the talk.

github.com/brettleehari/agent-crucible