Building an AI system to challenge radicalised thinking is one thing. Working out whether it’s actually doing any good is quite another.

Particularly when the people marking their own homework are also AI.

I built a multi-agent system to investigate whether conversational interventions could interrupt harmful ideological trajectories, testing different approaches across 180 simulated conversations with synthetic incel personas.

Then came the next challenge: how do you evaluate a shift in someone's thinking when all you have to go on is a conversation?

Enter two AI judges. One assessed signs of a softening stance. The other took a more sceptical view, looking for signs that hostility might persist beneath the apparent progress. When their scores diverged significantly, the system flagged the conversation for human review.

In this talk, I'll get under the bonnet of the evaluation system, explain how I designed the judges to challenge each other's conclusions, and explore what happens when they disagree. I'll also dig into the evaluators' own failure modes, and why surface-level politeness can be a very poor measure of ideological change.

Because if an AI system is going to attempt behavioural intervention, it had better be able to tell the difference between changing someone's mind and getting played.

What happens when you use AI to challenge extremist thinking and then ask another AI to judge if it worked? An inside look at multi-agent evaluation, dual-judge postures, and human-in-the-loop safety nets.