TL;DR
LLM leaderboards often overstate the performance of top models because ranking noisy benchmark scores rewards "sampling luck" alongside actual ability. This paper introduces RULER, a framework that uses a specialized bootstrapping technique to correct this selection bias and group models into statistically indistinguishable tiers, revealing that many "State-of-the-Art" (SOTA) claims are more fragile than they appear.
What Problem Does This Paper Address?
In the race for LLM supremacy, a fraction of a percentage point on a leaderboard like SWE-bench can determine which model is crowned "SOTA." However, these scores are just averages over a finite set of tasks. Because tasks are sampled, the scores contain noise.
When we rank models and pick the winner, we aren't just picking the most capable model; we are picking the model that got the "luckiest" with the specific tasks in that benchmark. This is a classic statistical trap known as the Winner’s Curse. The authors identify a broader phenomenon they call rank-induced bias: scores at the top of a leaderboard are systematically inflated, while those at the bottom are deflated. This makes it difficult for researchers and organizations to know if a new model is truly better or just benefited from a favorable draw of test cases.
Methods
The authors develop the Rank Uncertainty and Leaderboard Evaluation Report (RULER). The core engine of RULER is an $m$-out-of-$N$ bootstrap.
Standard bootstrapping (resampling $N$ items from $N$ with replacement) fails here because ranking is "non-smooth"—small changes in scores can lead to jumpy changes in rank, especially when models are closely tied. By resampling a smaller number of tasks ($m < N$), the authors can:
- Estimate Score Inflation: Calculate how much a model's rank-1 position is due to noise and subtract that "luck" from the reported score.
- Generate a Rank Probability Matrix (RPM): Show the probability of each model occupying every possible rank.
- Define Model Tiers: Instead of a rigid 1-to-N list, RULER groups models into tiers where their rank intervals overlap, indicating they are statistically indistinguishable.
Datasets
The authors audited 14 benchmarks across 5 major families to prove the ubiquity of the Winner's Curse:
- SWE-bench: (Verified, Bash-only, Multimodal, Multilingual, Pro) – Software engineering tasks.
- Tau2-Bench: (Airline, Retail, Telecom) – Agentic customer service scenarios.
- LiveBench: A contamination-free benchmark for reasoning and coding.
- AlpacaEval 2 & WildBench: Benchmarks based on human-style prompts and judge-based evaluation.
Key Insights and Ideas
- SOTA is often a statistical tie: On the popular SWE-bench Verified leaderboard, RULER found that the "top" tier actually contains 13 different models. The apparent winner only remains #1 in about 30.5% of resampled scenarios.
- The "Luck" Margin is Significant: In 2025, the top score on SWE-bench Verified rose 13 times. RULER found that 7 of those 13 increases were smaller than the estimated inflation from the Winner's Curse. In other words, many "new records" might just be noise.
- Bias is everywhere: The bias isn't just at Rank 1. It distorts the entire leaderboard. Models with identical true ability will appear to have different scores simply because they were sorted.
- Benchmark Design Guidance: The paper provides a formula for benchmark designers: to halve the expected winner inflation, you generally need to quadruple the number of tasks.
Key Results
The quantitative impact of RULER is striking. Across the 14 benchmarks, the adjustment for the #1 model ranged from 0.09 to 3.86 percentage points.
Using a "held-out test" (splitting benchmarks in half to see if the corrected score on one half predicts the raw score on the other), RULER reduced the average selection error at Rank 1 by 69.5%.
Limitations and Future Work
- Task Independence: The theoretical proofs assume tasks are independent and identically distributed (i.i.d.). While the authors show RULER is robust to some non-i.i.d. settings (like tasks coming from different GitHub repos), extreme correlations between tasks could still skew results.
- Data Contamination: RULER addresses sampling noise, but it cannot detect if a model has seen the test data during training (contamination) or if a vendor has over-optimized for a specific public benchmark.
- Residual Bias: In very small samples or when model differences are extremely tiny, the bootstrap can still leave some residual bias or occasionally overcorrect.
The authors advocate for RULER to be published alongside all major LLM leaderboards to provide a more honest picture of model performance and ranking precision.
Summary updated