Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

Introduction

As artificial intelligence models approach the "frontier" of human-level performance, the process of evaluating them has become a significant bottleneck. Human expert evaluation is slow, expensive, and difficult to scale. While the "LLM-as-judge" paradigm—using a powerful model like GPT-4 to evaluate others—offers a potential solution, it introduces significant biases. This paper investigates the theoretical and empirical limits of using LLMs for evaluation, particularly when the model being evaluated is as capable as, or more capable than, the judge itself.

The Evaluation Crisis and the Frontier

The machine learning community is currently facing an "evaluation crisis." The rapid release of models across diverse domains outpaces the availability of high-quality human annotations. To mitigate this, researchers often employ "model-as-judge" approaches (e.g., using GPT-4 to grade Llama-3). However, these judges are prone to biases, such as preferring their own outputs, favoring longer responses, or struggling with specific reasoning steps.

Techniques like Prediction Powered Inference (PPI) attempt to "debias" these judgments by using a small "gold standard" set of human labels to correct a larger pool of model-generated scores. This paper asks a fundamental question: how much human data can actually be saved through such methods? The authors focus on the "evaluation frontier," defined as the scenario where the evaluated model's accuracy ($b$) exceeds the judge's agreement rate with the truth ($AG$).

Methodology: Quantifying Efficiency

The authors develop a statistical framework modeling evaluation as a joint distribution between binary ground-truth scores ($s$) and proxy scores from a judge ($\tilde{s}$). They define a Sample Efficiency Factor ($\tau$), representing the ratio of "effective" data obtained relative to the amount of human data used. For example, $\tau=2$ implies that 100 human labels, when combined with model judgments, provide the statistical power of 200 human labels.

Using the Cramér-Rao bound, the authors demonstrate that the PPI estimator is near-optimal for this task. They also explore variations such as "stratified PPI," which groups tasks by difficulty, though they find this rarely breaks the fundamental efficiency barriers in practice.

The Factor of Two Ceiling

The study's most significant theoretical finding is the "Factor of Two" ceiling. At the evaluation frontier, no matter the volume of cheap LLM judgments available, statistical power cannot be improved by more than a factor of two ($\tau \le 2$). This limit is fundamentally tied to the Pearson correlation between the judge and the ground truth. The authors prove in Theorem 6 and Corollary 7 that when a judge is "weaker" than the subject ($0.5 \le AG \le b$), this correlation is mathematically capped at $\sqrt{0.5}$, leading directly to the $\tau \le 2$ limit.

Rank scrambling at the frontier: LLM judges (GPT-4 and Llama-3 70B) fail to preserve true model rankings when evaluating models of similar or higher capability.

The Judge Accuracy Trap

A common assumption is that a judge with high agreement (e.g., 85%) with humans is sufficient. However, the paper shows that even high-agreement judges can completely scramble the rankings of top-tier models if the judge is less capable than the models it evaluates. As shown in the figure above, rankings generated by GPT-4 and Llama-3 70B labels diverge significantly from true labels when applied to frontier models. This underscores that agreement does not equal utility; a judge may agree with humans on easy cases but fail to provide signal where frontier models actually differ.

Empirical Evidence

The theoretical limits were validated across several standard benchmarks, including MMLU (Massive Multitask Language Understanding), MT-Bench, HELM (Holistic Evaluation of Language Models), and TruthfulQA. In tests on MMLU, the sample efficiency factor $\tau$ consistently remained below 2 for frontier models. The only instances where $\tau$ exceeded 2 occurred when a significantly stronger model (e.g., GPT-4) was used to judge a much weaker, older model (e.g., Llama-2-7b). Even when using "soft" labels (model probabilities instead of binary scores), the "factor of two" limit remained remarkably robust.

Conclusion and Future Work

The findings suggest that while LLM-based evaluation and debiasing techniques are useful, they are not a silver bullet for scaling evaluation at the frontier. We are effectively capped at "twice the data." Future work may explore whether moving beyond black-box evaluations—such as analyzing internal model states or using multi-step reasoning chains for judges—can break this $\tau \le 2$ limit by improving the correlation with ground truth. Additionally, if the task of evaluating is significantly easier than the task itself (e.g., verifying a math solution), judges might perform better, potentially allowing for higher savings.

Summary updated

Abstract

High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.
BibTeX
@inproceedings{dorner2025limits,
  author = {Florian E. Dorner and Vivian Y. Nastl and Moritz Hardt},
  title = {{Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=NO6Tv6QcDs},
}
Figure