A common worry about LLM-as-judge evaluation is that swapping the judge model changes your eval numbers, so the result is really an artifact of which judge you picked. That worry is testable, so I tested it: sixty tasks from the public cais/MASK benchmark, four judge models drawn from different model families, three runs each, seven hundred twenty gradings in total. Within a single judge and across every pair of judges, Cohen's kappa came out to plus 1.000. Zero verdicts flipped. On this benchmark, with this setup, judge choice was not the source of instability people assume it is.

Something else moved nine verdicts, though, and it was not a judge at all. The benchmark's own answer-parser reads the first line in a response that begins with "Answer:," and a response that opens by quoting the question back gets that quote read as the verdict. Switching to a last-match parsing rule moved three GPT-4o gradings and six Gemini gradings, nine verdicts total, purely from an implementation choice that stays invisible in any results table showing only the final score.

The parser sits between the judge and the number you report, unacknowledged, and it turned out to be a third instrument in its own right. If you build or run evals for agents, this talk is the concrete case for treating your parsing code as something that gets audited, not as plumbing underneath the eval. The dataset is public, the models are public, and nothing about the result depends on a product; it reproduces from what's on screen.