I spent three months making our RAG system better. Then I actually measured it and found I'd mostly been shipping noise.

Some context: we'd built a knowledge layer over a large Python monorepo, the kind of thing that answers "which service throws this error code" for engineers. Retrieval metrics looked fine. Dashboard was green. Users kept telling us the answers were wrong.

It wasn't the model or the chunking. Our evaluation just couldn't tell a real improvement apart from a coin flip — run it twice on the same code, get two different scores. We'd promoted a change on a 3-point gain that turned out to sit inside the error bar.

So in this session I'll build a small context layer live, then break it in front of you. Bad eval, bad benchmark, a judge that scores the wrong answer higher than the right one, and a document I poison halfway through that the pipeline never notices. Some of this is embarrassing — a stronger model beat our benchmark by 16 points and got measurably worse on real user traffic.

Repo goes out at the end so you can run all of it yourself.

Takeaways
• Why your eval probably can't detect the improvement you just shipped
• A cheap way to get error bars on it without tripling what you spend on evals
• The boring governance work that moved our numbers more than any amount of
retrieval tuning did

Format Session — 40 minutes (incl. Q&A)
Tags Gen AI, LLM · Agents, RAG · Data