Introduction
Evaluating Large Language Models (LLMs) has increasingly shifted toward human preference ratings, exemplified by platforms like Chatbot Arena. While these leaderboards provide valuable insights, they typically rely on the Bradley-Terry model and Maximum Likelihood Estimation (MLE), which suffer from three critical limitations. First, they exhibit bias blindness, failing to account for known judge tendencies such as length bias (preferring longer answers) or position bias. Second, they are cost-prohibitive, requiring massive amounts of human annotation to reach statistical significance. Finally, they suffer from shift-invariance, meaning ratings are isolated by task; a model's score in "Coding" cannot be meaningfully compared to its score in "Math" because the scales are independent.
To address these challenges, we introduce Polyrating, a multivariate rating system that incorporates judge biases and leverages informative priors to significantly improve sample efficiency and cross-task comparability.
Methodology
Polyrating moves away from the standard univariate approach toward a multivariate linear model fitted via Maximum A Posteriori (MAP) estimation. This framework introduces several technical innovations designed to stabilize and enrich the rating process.
Key Technical Innovations
- Shared Bias Parameters ($\alpha$): Polyrating models features like answer length and readability as shared parameters across all models. This allows the system to quantify exactly how many "rating points" a specific bias contributes to a model's total score, effectively "de-biasing" the core performance metric.
- Task-Specific Modifiers ($\beta$): Instead of fitting entirely new ratings for every task, Polyrating starts with a base rating ($R_{base}$) for each model and adds task-specific modifiers. This hierarchical structure allows for more stable estimation in data-scarce scenarios.
- Informative Priors: By utilizing MAP estimation, Polyrating can be "warmed up" using cheaper data sources, such as LLM-based judges or traditional benchmarks (e.g., MixEval). These priors regularize the model, breaking the shift-invariance problem and enabling the creation of truly comparable cross-task leaderboards.
Experimental Setup
The system was validated using three large-scale datasets: Chatbot Arena (over 1 million human-judged comparisons), Wildbench (for LLM-based judge evaluations), and MixEval-Hard (to demonstrate how classical benchmark scores can improve human rating efficiency).
Key Insights and Results
Quantifying Judge Bias
Polyrating reveals the significant impact of the "Length Tax." In Chatbot Arena, simply providing a longer response boosts a model's rating by approximately 41 points for human judges and 48 points for LLM judges. Interestingly, the system also identified a divergence in judge preferences: while humans prefer more readable text (adding ~11 points), LLM judges prefer denser text, actually penalizing readability by approximately 4 points.
Sample Efficiency and Convergence
One of the most significant advantages of Polyrating is its cost efficiency. By leveraging priors and shared parameters, the system improves sample efficiency by 38% to 77% for new tasks and up to 41% for new models. This translates to massive savings in human annotation budgets.
As shown in the convergence analysis, Polyrating (orange) reaches a lower logistic loss much faster than the univariate baseline (blue) across various domains, including Coding and Hard Prompts. This faster convergence allows researchers to achieve reliable rankings with a fraction of the data previously required.
Multivariate Leaderboards
The resulting leaderboards provide a more nuanced view of model capabilities. For instance, while top-tier models like GPT-4o remain strong all-rounders, Polyrating highlights that models from Chinese providers (such as Yi or Qwen) show statistically significant gains specifically in Chinese task modifiers—nuances that were previously obscured by the shift-invariance of standard Elo systems.
Conclusion and Future Work
Polyrating represents a significant step toward more robust and affordable LLM evaluation. By explicitly modeling bias and utilizing informative priors, it provides a more accurate reflection of model capability while reducing the reliance on expensive human labels. Future work will focus on the automatic discovery of relevant bias dimensions to reduce the need for manual feature engineering and exploring how to move from relative preference to absolute performance metrics.
Summary updated