Prediction-Powered Ranking of Large Language Models

Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, Manuel Gomez Rodriguez

TL;DR

Ranking Large Language Models (LLMs) using "LLM-as-a-judge" is cost-effective but statistically risky due to unquantified biases. This paper introduces a framework that combines a small set of human labels with a large set of LLM-judge labels to produce "rank-sets"—ranges of possible ranking positions—that are mathematically guaranteed to contain the true human-aligned rank with high probability.

What Problem Does This Paper Address?

Evaluating LLMs is notoriously difficult because many tasks (like creative writing or coding assistance) don't have a single "correct" answer. The industry standard has shifted toward human preference alignment: Model A is better than Model B if humans prefer its output more often.

Platforms like the LMSYS Chatbot Arena use pairwise comparisons to generate Elo ratings or win-rates. However, human evaluation is slow and expensive. To scale, researchers often use "strong" LLMs (like GPT-4) to act as judges. The problem? LLM judges aren't perfect proxies for humans. They have their own biases, and until now, there was no rigorous way to measure how much uncertainty these biases—or the limited number of samples—introduce into the final leaderboard.

Methods

The authors propose a statistical framework called Prediction-Powered Ranking (PPR). It builds on the recently developed theory of Prediction-Powered Inference (PPI).

  1. Win-Rate Metric: Instead of Elo, which is sensitive to the order of games, they use the win-rate—the probability a model is preferred over a competitor picked at random.
  2. Rank-Sets: Rather than assigning a single rank (e.g., "Model X is #3"), the framework outputs a set of possible positions (e.g., "Model X is between #2 and #5").
  3. Hybrid Inference: The core innovation is using a small "gold standard" dataset (human comparisons) to correct a much larger "noisy" dataset (LLM judge comparisons).
  4. Confidence Ellipsoids: The algorithm constructs a joint confidence region for the win-rates of all models. By calculating the geometric distance between this ellipsoid and the hyperplanes where two models would have equal win-rates, the framework determines which ranking positions are statistically plausible.

Importantly, the framework provides asymptotic coverage guarantees: with a user-defined confidence level (e.g., 95%), the resulting rank-sets are guaranteed to contain the true ranking consistent with human preferences.

Datasets

  • LMSYS Chatbot Arena: The primary real-world dataset, consisting of 33,481 human pairwise comparisons across 20 different LLMs.
  • LLM Judges: The authors generated their own judge labels for the same prompts using GPT-4, GPT-3.5, and Claude 3 Opus.
  • Synthetic Data: Used to validate the theoretical guarantees in a controlled environment where the "ground truth" win-rates are known.

Key Insights and Ideas

  • LLM Judges are Unreliable Alone: The study finds that rankings constructed only by LLM judges often diverge significantly from human rankings. As shown in the paper, the top-performing rank for 7 out of 12 models differed between the LLM judge and the human baseline.
  • The "De-biasing" Effect: You don't need a massive amount of human data to get a reliable ranking. By using a small human sample to "anchor" a large LLM-judged dataset, PPR achieves a much better trade-off between precision (small rank-sets) and accuracy (covering the true rank).
  • Quantifying "I Don't Know": Large rank-sets serve as a vital signal to practitioners that the current data is insufficient to distinguish between certain models, preventing overconfident (and potentially false) claims about model superiority.

Key Results

The authors compared PPR against "LLM-only" and "Human-only" baselines.

Figure 1: Average rank-set size against baseline intersection probability for rank-sets constructed using only pairwise comparisons by a strong LLM (Llm Gpt4, Llm Gpt3.5 and Llm Cl3), only pairwise comparisons by humans (Human Only), and pairwise comparisons by both a strong LLM and humans (Ppr Gpt4, Ppr Gpt3.5 and Ppr Cl3) for different values of α and n = 990. Smaller (larger) average rank-set sizes and larger (smaller) intersection probabilities are better (worse). In all panels, 95% confidence bars for the rank-set size are not shown, as they are always below 0.02.

As seen in Figure 1 above, LLM-only rankings (the clusters on the far left) have extremely low "baseline intersection probability," meaning they frequently fail to overlap with the human-validated ranking. PPR methods (the shapes on the right) successfully bridge this gap, maintaining high alignment even with relatively small rank-sets.

Figure 2: Average rank-set size against baseline intersection probability for rank-sets constructed using pairwise comparisons by both a strong LLM and humans for different values of n and α. Smaller (larger) average rank-set sizes and larger (smaller) intersection probabilities are better (worse). In all panels, 95% confidence bars for the rank-set size are not shown, as they are always below 0.04.

Figure 2 demonstrates that as the number of human comparisons ($n$) increases, the average rank-set size shrinks rapidly, providing more precise rankings while maintaining the statistical coverage guarantee. Among the judges, GPT-4 was found to be the most aligned with humans, though it still required PPR correction to be reliable.

Limitations and Future Work

  • I.I.D. Assumption: The framework assumes queries and comparisons are drawn from a fixed distribution. It does not yet account for "distribution shift" (e.g., if the types of prompts change over time).
  • Truthfulness: It assumes human labels are the ground truth. It doesn't account for adversarial humans or strategic voting intended to boost specific models.
  • Computational Scaling: For a very large number of models ($k$), the $O(k^2)$ pairwise comparisons required for win-rates might become a bottleneck.
  • Finite Samples: The coverage guarantees are asymptotic; the authors suggest that developing "PAC-style" finite-sample guarantees would be a valuable next step.

Summary updated

Abstract

Large language models are often ranked according to their level of alignment with human preferences -- a model is better than other models if its outputs are more frequently preferred by humans. One of the popular ways to elicit human preferences utilizes pairwise comparisons between the outputs provided by different models to the same inputs. However, since gathering pairwise comparisons by humans is costly and time-consuming, it has become a common practice to gather pairwise comparisons by a strong large language model -- a model strongly aligned with human preferences. Surprisingly, practitioners cannot currently measure the uncertainty that any mismatch between human and model preferences may introduce in the constructed rankings. In this work, we develop a statistical framework to bridge this gap. Given a (small) set of pairwise comparisons by humans and a large set of pairwise comparisons by a model, our framework provides a rank-set -- a set of possible ranking positions -- for each of the models under comparison. Moreover, it guarantees that, with a probability greater than or equal to a user-specified value, the rank-sets cover the true ranking consistent with the distribution of human pairwise preferences asymptotically. Using pairwise comparisons made by humans in the LMSYS Chatbot Arena platform and pairwise comparisons made by three strong large language models, we empirically demonstrate the effectivity of our framework and show that the rank-sets constructed using only pairwise comparisons by the strong large language models are often inconsistent with (the distribution of) human pairwise preferences.
BibTeX
@inproceedings{chatzi2024prediction,
  author = {Ivi Chatzi and Eleni Straitouri and Suhas Thejaswi and Manuel Gomez Rodriguez},
  title = {{Prediction-Powered Ranking of Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=7V62sQ5Jra},
}
Figure