Why your AI benchmark’s judge model may be rigging the results

A software engineer recently ran a controlled test comparing five large language models on a set of ten straightforward questions and ten more deliberately tricky ones. The goal was to measure cost per query, response speed, and answer quality across a free local model and four paid services: GPT, DeepSeek, and two versions of Claude. The setup was simple in principle: run the same prompts through each model twice, record the outputs, and score the answers using a second model. The entire experiment cost about 21 cents, a small price that unexpectedly changed how the results were interpreted.

At first glance, the quality scores looked decisive. All five models clustered tightly between 0.92 and 0.97 on a 0-to-1 scale, with the cheapest and fastest model performing as well as the most expensive one. The pricier Sonnet model produced longer answers, 239 tokens compared to GPT’s 65, but did not score higher. The initial conclusion might have been to recommend the cheaper option, but the engineer knew better than to trust a tidy ranking built on tiny gaps. To test whether the differences were real, the developer calculated confidence intervals around each score and around the gaps between models. Every paid model’s interval overlapped with every other, meaning the quality differences were statistically indistinguishable. In one earlier run, GPT had ranked first; in this run, it ranked third. The order had shifted, but the tie remained. The ranking was noise, not signal.

The deeper problem was the judge itself. The quality scores came from a free local Llama model, which was also one of the five models being evaluated. A contestant was grading its own competition, and the judge was not up to the task. When the engineer reran the grading using a stronger paid judge instead, the results changed dramatically. Every paid model’s score increased, while the local Llama’s score dropped. Under the weaker judge, the local model had appeared only slightly behind the pack, scoring 0.936 compared to Sonnet’s 0.97. With the better judge, the gap widened: the local model passed 35 out of 40 answers, while every paid model passed all 40. The cheap judge had been softening the results for the weaker model, effectively masking its mistakes.

The discrepancies were not random. On the harder set of questions, designed to expose weaknesses, the cheap judge repeatedly passed answers that were clearly wrong. In one case, an answer incorrectly stated the number of planets in the solar system and included a list that contradicted itself. The cheap judge approved both versions, praising one for mentioning Pluto’s reclassification and the other for omitting it, depending solely on which answer appeared first. The same bias showed up in pairwise comparisons, where the judge was asked to pick the better of two answers to the same question. On easy questions, the cheap judge changed its mind 45% of the time when the order of the answers was swapped, compared to 19% for the paid judge. When it flipped, it chose the answer it saw first 93% of the time, indicating it was rewarding position rather than quality.

The engineer also discovered that the cheap judge favored its own answers when grading a group that included its outputs. Despite scoring last when judged independently, the local model ranked third in the group, suggesting favoritism. The bias was not subtle: the judge was systematically rewarding weaker models and its own responses, undermining the entire benchmark. The lesson was clear. A benchmark is only as reliable as the judge that scores it, and using an underpowered model to grade others can produce a scoreboard that declares real mistakes as correct answers.

The experiment also highlighted the risks of treating benchmarking as a cost-saving exercise. When every API call incurs a small fee, there is pressure to minimize testing: run fewer passes, use cheaper judges, or skip replications. In this case, the second pass of each question was initially seen as wasteful, but it turned out to be essential. Three of the five models scored 19 out of 20 instead of a perfect 20, revealing that some answers hovered right at the pass threshold and flipped between passes. A single pass would have recorded a coin flip as a definitive result. The money spent on the second pass exposed the truth that cheaper testing would have hidden.

Beyond the immediate findings, the project underscored the importance of reproducibility in AI evaluation. Early in the experiment, the engineer saved the judge’s verdicts but not its reasoning. When later attempts to verify specific claims required rerunning the judge, the outputs changed because the models are non-deterministic. The saved verdicts no longer matched the new answers, rendering the evidence unreliable. The solution was to rebuild every quote and number from the saved outputs using a small generator, ensuring that the proof and the report always aligned. Evidence that drifts is not evidence at all.

The takeaway is not about which model is best but about how benchmarks should be designed. Raw scores that differ by hundredths of a point are often meaningless; small gaps can vanish on a rerun. Before trusting any ranking, it is critical to estimate how much the scores would shift with a new run. In this case, allowing for that swing collapsed the apparent differences, revealing that the paid models were effectively tied. The ranking was an artifact of noise, not a reflection of true performance. The experiment also reinforced a simple rule: never let the contestant grade its own competition. The judge must be strong enough to catch the mistakes that matter, especially on the hard questions designed to expose weaknesses.

This was the fifth and final project in a series focused on testing AI systems, covering evaluation harnesses, retrieval-augmented generation, red-teaming, and agent testing. This benchmark closed the loop by showing that the judge used to grade all the others must itself be scrutinized first. The next step is not another tool but a reflection on what these five projects taught about working with models, not just testing them, and whether those lessons hold up under real-world use.