SOTA or Luck? The Winner's Curse in LLM Leaderboards

Sikun Xu, Raphael Thomadsen, Dennis J. Zhang, Heng ZhangarXiv 2026 · arXiv

TL;DR

LLM leaderboards often overstate the performance of top models because ranking noisy benchmark scores rewards "sampling luck" alongside actual ability. This paper introduces RULER, a framework that uses a specialized bootstrapping technique to correct this selection bias and group models into statistically indistinguishable tiers, revealing that many "State-of-the-Art" (SOTA) claims are more fragile than they appear.

What Problem Does This Paper Address?

In the race for LLM supremacy, a fraction of a percentage point on a leaderboard like SWE-bench can determine which model is crowned "SOTA." However, these scores are just averages over a finite set of tasks. Because tasks are sampled, the scores contain noise.

When we rank models and pick the winner, we aren't just picking the most capable model; we are picking the model that got the "luckiest" with the specific tasks in that benchmark. This is a classic statistical trap known as the Winner’s Curse. The authors identify a broader phenomenon they call rank-induced bias: scores at the top of a leaderboard are systematically inflated, while those at the bottom are deflated. This makes it difficult for researchers and organizations to know if a new model is truly better or just benefited from a favorable draw of test cases.

Methods

The authors develop the Rank Uncertainty and Leaderboard Evaluation Report (RULER). The core engine of RULER is an $m$-out-of-$N$ bootstrap.

Standard bootstrapping (resampling $N$ items from $N$ with replacement) fails here because ranking is "non-smooth"—small changes in scores can lead to jumpy changes in rank, especially when models are closely tied. By resampling a smaller number of tasks ($m < N$), the authors can:

  1. Estimate Score Inflation: Calculate how much a model's rank-1 position is due to noise and subtract that "luck" from the reported score.
  2. Generate a Rank Probability Matrix (RPM): Show the probability of each model occupying every possible rank.
  3. Define Model Tiers: Instead of a rigid 1-to-N list, RULER groups models into tiers where their rank intervals overlap, indicating they are statistically indistinguishable.

Datasets

The authors audited 14 benchmarks across 5 major families to prove the ubiquity of the Winner's Curse:

  • SWE-bench: (Verified, Bash-only, Multimodal, Multilingual, Pro) – Software engineering tasks.
  • Tau2-Bench: (Airline, Retail, Telecom) – Agentic customer service scenarios.
  • LiveBench: A contamination-free benchmark for reasoning and coding.
  • AlpacaEval 2 & WildBench: Benchmarks based on human-style prompts and judge-based evaluation.

Key Insights and Ideas

  • SOTA is often a statistical tie: On the popular SWE-bench Verified leaderboard, RULER found that the "top" tier actually contains 13 different models. The apparent winner only remains #1 in about 30.5% of resampled scenarios.
  • The "Luck" Margin is Significant: In 2025, the top score on SWE-bench Verified rose 13 times. RULER found that 7 of those 13 increases were smaller than the estimated inflation from the Winner's Curse. In other words, many "new records" might just be noise.
  • Bias is everywhere: The bias isn't just at Rank 1. It distorts the entire leaderboard. Models with identical true ability will appear to have different scores simply because they were sorted.
  • Benchmark Design Guidance: The paper provides a formula for benchmark designers: to halve the expected winner inflation, you generally need to quadruple the number of tasks.

Key Results

The quantitative impact of RULER is striking. Across the 14 benchmarks, the adjustment for the #1 model ranged from 0.09 to 3.86 percentage points.

Figure 1: The winner on SWE-bench Verified remains statistically unresolved.

Using a "held-out test" (splitting benchmarks in half to see if the corrected score on one half predicts the raw score on the other), RULER reduced the average selection error at Rank 1 by 69.5%.

Table 1: The correction reduces average selection error across 14 benchmarks.

Limitations and Future Work

  • Task Independence: The theoretical proofs assume tasks are independent and identically distributed (i.i.d.). While the authors show RULER is robust to some non-i.i.d. settings (like tasks coming from different GitHub repos), extreme correlations between tasks could still skew results.
  • Data Contamination: RULER addresses sampling noise, but it cannot detect if a model has seen the test data during training (contamination) or if a vendor has over-optimized for a specific public benchmark.
  • Residual Bias: In very small samples or when model differences are extremely tiny, the bootstrap can still leave some residual bias or occasionally overcorrect.

The authors advocate for RULER to be published alongside all major LLM leaderboards to provide a more honest picture of model performance and ranking precision.

Summary updated

Abstract

Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an $m$-out-of-$N$ bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces $Θ(N^{-1/2})$ ranking bias to expected residual bias $o(N^{-1/2})$. Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.
BibTeX
@misc{xu2026sota,
  author = {Sikun Xu and Raphael Thomadsen and Dennis J. Zhang and Heng Zhang},
  title = {{SOTA or Luck? The Winner's Curse in LLM Leaderboards}},
  year = {2026},
  eprint = {2611.22227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22227},
}
Figure