Only about a quarter of internet users are native English speakers, yet most AI systems are built and evaluated as if that number were 100%. For organizations in regulated and public-facing industries, this creates a trust problem.

This session shares findings from "Lost in Translation: An Autonomous AI and Language Translation Study," a Women in AI research project led by Karen Jensen. Six AI models were evaluated in Hindi and Chinese by native-speaker reviewers: I led the Hindi evaluation and Charlotte Tao led the Chinese evaluation. Each model was scored across five dimensions: localization fluency, semantic fidelity, linguistic accuracy, terminology precision, and cultural nuance. No model excelled across all dimensions in either language and the only open-source model used scored lowest in both.

The most concerning finding was bias laundering. A Forward and Reverse Translation Audit showed that bias introduced when translating into a language can disappear when the text is translated back into English making it invisible to English-only reviewers. The study also found a consistent gap between how models rated their own translations and how they actually performed. In my blind Hindi review, five of six models scored the lowest rating on localization fluency, one silently dropped an entire section and technical terms were often left in English. Yet every model preserved gendered references correctly, which I didn't expected for Hindi, so it was a reminder that the risks we expect are not always the ones we find.

Drawing on this study and on my experience building multilingual agentic AI at enterprise scale across 17+ languages, I will share why native speakers must be part of evaluation, why model self-assessment and back-translation are not enough and how to design human-in-the-loop review that catches what automated checks miss.

Key takeaways:
- What bias laundering is and why English-only review cannot detect it
- Why model self-assessment is not a reliable evaluation signal
- A five-dimension framework for evaluating multilingual AI output
- How to build native-speaker review into AI governance without multiplying cost