TL;DR
This paper introduces "Consistency Score," a novel metric that measures how consistently an LLM judges pairwise comparisons between other LLMs. This score strongly correlates (0.91 Pearson correlation) with established Elo scores, offering a cheap and scalable alternative for evaluating LLM intelligence without human input.
What Problem Does This Paper Address?
The rapid proliferation of Large Language Models (LLMs) has created a significant challenge in evaluating their performance. While human-based evaluation, particularly through methods like the LMSYS Chatbot Arena, provides a gold standard (Elo scores), it's expensive and doesn't scale with the increasing number of models. LLM-as-judge approaches have emerged but suffer from biases (positional, verbosity, self-enhancement) and often show poor alignment with human preferences. Existing objective benchmarks, while useful for domain-specific knowledge, fail to capture nuanced aspects like style and complex task performance. The paper aims to bridge this gap by proposing a scalable, cost-effective, and human-free proxy for LLM intelligence that correlates well with human-derived Elo scores.
Methods
The core innovation is the Consistency Score. This metric quantifies how consistently an LLM judge selects a preferred response when presented with pairwise comparisons of other LLMs.
- Preference Elicitation: The authors employ a two-stage chain-of-thought prompting strategy. First, the judge LLM is given a user question and two model responses, along with evaluation criteria. It generates a paragraph of reasoning for its preference. Second, it uses this reasoning to explicitly select a preference (model 1, model 2, or a tie). This two-stage approach helps mitigate LLM biases and ensures better adherence to evaluation criteria.
- Consistency Calculation:
- For each matchup between two models ($m_i$ and $m_j$), the probability of $m_i$ winning ($p_{ij}$) is calculated based on the number of contests, wins, and ties.
- The variance for this matchup is calculated as $p_{ij} \cdot (1 - p_{ij})$. This variance is inversely correlated with consistency: lower variance means higher consistency.
- The average variance across all matchups for a judge LLM is computed.
- This average variance is then transformed into the Consistency Score using the formula: $1 - 4 \cdot \text{Average Variance}$. This score ranges from 0 (random judge) to 1 (perfectly consistent judge).
The method is designed to be independent of human evaluation, prior knowledge, and traditional data leakage issues.
Datasets
The primary dataset used for evaluation is the arena-human-preference-55k dataset (Chiang et al., 2024). This dataset contains unique prompts, paired responses from competing LLMs, and the identities of those LLMs. Crucially, the authors did not use the human preference annotations from this dataset, focusing solely on using the LLM responses to generate contests evaluated by their judge LLMs. They extracted a subset of 2800 question-response pairs involving 35 different LLMs across 140 matchups.
Key Insights and Ideas
- Consistency as a Proxy: The fundamental insight is that an LLM's ability to consistently judge the quality of other LLMs' outputs is a strong indicator of its own intelligence, as measured by Elo scores.
- LLM Judges Can Be Reliable (with caveats): While LLM judges have known biases, their consistency in applying judgment criteria can be a robust signal, even if the absolute judgment isn't perfectly aligned with human preference.
- Data Matters for Consistency: Matchups between models with significantly different Elo scores are more effective for identifying consistent judges. This is because such pairs provide clearer signals for differentiation, allowing strong judges to shine and weak judges to reveal their inconsistencies.
- Convergence with Few High-Difference Matchups: A surprisingly small number of matchups (specifically, the 10 with the highest Elo differences) can achieve a significant portion of the correlation with Elo scores, highlighting the efficiency of this approach.
Key Results
- The proposed Consistency Score achieved a Pearson correlation coefficient of 0.91 with LMSYS Elo scores across 24 evaluated LLMs.
- Predicting an LLM's Elo score based on its Consistency score resulted in a mean absolute error of 35.2 Elo points.
- The metric is effective across a diverse range of LLMs, with strong correlations observed even within clusters of lower-performing models.
- Using just the 10 matchups with the highest Elo differences yielded a correlation of 0.88, demonstrating the power of carefully selected data.
Limitations and Future Work
Limitations:
- Distinguishing Top Models: The Consistency Score struggles to differentiate between models that are very close in Elo score, particularly among the highest-performing cluster (Pearson correlation of 0.12 for the highest cluster).
- Noisy Elo Scores: The underlying Elo scores themselves are inherently noisy and subjective, which can affect the precision of the Consistency Score's correlation.
Future Work:
- Optimized Contest Sets: Curating dedicated contest sets specifically for information extraction, rather than using general-purpose data, could further enhance performance.
- Weighting Formula: Developing a weighting formula that accounts for Elo differences in matchups could improve robustness, potentially downweighting noisier comparisons between closely matched models.
- Prompt Engineering: Advanced prompt engineering might unlock better performance from stronger models, especially if current instructions are imprecise.
- Sophisticated Uncertainty Modeling: Incorporating patterns where consistently high-performing models across diverse contests demonstrate greater reliability could lead to more robust uncertainty models.
Summary updated