Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

TL;DR

Evaluating frontier language models using weaker LLMs as judges introduces severe biases that distort model rankings. While statistical debiasing methods can mitigate this by pairing a large number of cheap judge predictions with a small sample of expensive ground-truth labels, this paper proves a sobering theoretical limit: when the judge is less accurate than the model being evaluated, no debiasing method can reduce the required ground-truth data by more than half.

What Problem Does This Paper Address?

As large language models (LLMs) rapidly advance, evaluating them via human experts has become an expensive, slow, and unsustainable bottleneck. To scale evaluation, researchers frequently employ the "LLM-as-a-judge" paradigm, using a powerful existing model to score the outputs of new models.

However, LLM judges suffer from systemic biases—such as favoring their own outputs, preferring longer responses, or exhibiting choice-order bias—which can severely disrupt benchmark rankings. To fix this, an emerging class of debiasing techniques (like Prediction Powered Inference, or PPI) uses a small set of high-quality ground-truth labels to mathematically correct the biases of a large pool of cheap LLM judgments.

This paper investigates a critical question at the "evaluation frontier": How effective are these debiasing methods when we try to evaluate a newly released model that is actually better than the judge model itself?

Figure 1: Model ranks on MMLU based on true labels compared to LLM labels in a semi-synthetic setting (a) and for the top-10 models on HELM by July 2024 (b). Using LLM labels heavily perturbs the ranking despite high judge accuracy.

Methods

The authors formalize the evaluation problem as an estimation of a model's expected binary score (e.g., correct/incorrect classification, safe/unsafe output) over a prompt distribution. They treat the ground-truth scoring function and the judge's proxy scoring function as black boxes, focusing purely on their joint distribution.

To determine the absolute limits of debiasing, the authors apply the multivariate Cramér-Rao bound to establish a fundamental variance floor for any unbiased estimator that combines a small number of ground-truth labels ($n$) with a large number of proxy judge labels ($N$). They define a metric called the sample efficiency factor ($\tau$), which quantifies the effective multiplier on your ground-truth data size achieved by incorporating the judge.

They analyze both hard binary proxy judgments and "soft" continuous proxy judgments (where the judge provides its internal next-token probabilities rather than a hard decision).

Datasets

The theoretical limits were empirically validated using model outputs and judgments on three popular benchmarks:

  • MMLU (Massive Multitask Language Understanding): Multiple-choice question answering, using model predictions sourced from the HELM leaderboard.
  • MT-Bench: An arena-style chatbot evaluation benchmark designed to test instruction-following capabilities.
  • TruthfulQA: A benchmark measuring whether models mimic human falsehoods and conspiracy theories.

Flagship models like GPT-4, Claude 3.5 Sonnet, and LLaMa-3-70B were cross-evaluated as both subjects and judges.

Key Insights and Ideas

  • The "Frontier Barrier" ($\tau \le 2$): The core mathematical takeaway is that if a judge model's agreement rate with the ground truth is lower than the accuracy of the model being evaluated, the squared Pearson correlation between the judge and the ground truth is strictly capped at 0.5. Consequently, the maximum sample efficiency factor is 2. You can never beat simply doubling your human-annotated data.
  • High Agreement is a Mirage: A common justification for using LLM judges is that they achieve high agreement (e.g., 80-85%) with humans. The authors prove that high agreement does not prevent severe ranking distortions, nor does it guarantee significant sample efficiency gains.
  • Soft Labels Help, But Don't Break the Barrier: Transitioning from binary judge decisions to continuous probability scores ("soft agreement") does improve data efficiency in practice, but it still respects the fundamental theoretical limit when evaluating superior frontier models.

Key Results

  • Theorem 5 & 6: The authors mathematically prove that the optimized Prediction Powered Inference (PPI) estimator is near-optimal for non-adaptive binary evaluation, and that under the frontier assumption ($0.5 \le AG(m) \le b(m)$), the sample efficiency factor $\tau$ cannot exceed 2.
  • Empirical Confirmation: In experiments on MMLU and MT-Bench, when current state-of-the-art models were evaluated by weaker judges, the empirical sample efficiency gains consistently stayed below 2. For instance, using LLaMa-3-70B to judge Claude 3.5 Sonnet yielded a sample efficiency factor well below 1.5, offering very modest economic savings in practice.

Limitations and Future Work

The paper highlights a few boundaries to its negative results:

  1. Uniform Sampling Assumption: The proofs assume uniform data sampling. More advanced techniques like stratified PPI might yield slightly better results by breaking the assumptions on a per-stratum basis, though early experiments suggest gains remain modest.
  2. Task Difficulty Symmetry: The bound assumes that performing the task and judging the task are of similar difficulty. If a task is incredibly hard to generate but trivial to evaluate (a massive asymmetry), an LLM judge might break through the factor-of-two limit. Exploring these asymmetric evaluation tasks is a promising avenue for future research.

Summary updated

Abstract

High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.
BibTeX
@inproceedings{dorner2025limits,
  author = {Florian E. Dorner and Vivian Y. Nastl and Moritz Hardt},
  title = {{Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=NO6Tv6QcDs},
}
Figure