TL;DR
Current methods for evaluating Large Language Models (LLMs) often fall into the trap of "style over substance," where judges—both human and AI—prefer long, well-written responses even if they contain factual errors. This paper introduces the Multi-Elo Rating System (MERS), which improves evaluation reliability by forcing judges to independently rate models across three distinct dimensions: Accuracy, Helpfulness, and Language.
What Problem Does This Paper Address?
As LLMs become more sophisticated, traditional benchmarks (like multiple-choice tests) fail to capture their performance on open-ended tasks. The industry has pivoted to "Elo ratings," where judges (either humans or stronger LLMs like GPT-4) compare two model outputs and pick a winner.
However, this "black box" approach to judging is prone to significant biases. We don't actually know if a judge chooses an answer because it is factually correct, or simply because it is long and uses professional-sounding grammar. The authors set out to quantify these biases and determine if we can trust crowd-sourced workers or even other LLMs to be objective referees.
Methods
The researchers conducted a controlled experiment by curating a dataset of intentionally flawed answers. They used GPT-4 to generate responses to 40 general-purpose questions, but with specific "bugs" introduced:
- Factual Errors: Ranging from "one minor error" to "several major errors."
- Language Errors: Simulating an "advanced learner" (spelling mistakes) or an "intermediate learner" (grammatical issues).
- Length Variations: Comparing "Long" (~100 words) vs. "Short" (~50 words) versions.
They then had four groups of judges evaluate these outputs: crowd-sourced workers (Amazon Mechanical Turk), expert annotators (holding Master’s degrees), GPT-4, and Claude-1.
To fix the biases they discovered, they proposed MERS, which breaks the evaluation into three independent prompts:
- Accuracy: Focuses on factual correctness and logic.
- Helpfulness: Focuses on relevance and depth.
- Language: Focuses on clarity, grammar, and tone.
Datasets
The study utilized a subset of the Vicuna benchmark (Chiang et al., 2023), specifically 40 questions across categories like "common sense," "knowledge," and "counterfactual." They excluded math and coding questions to ensure that generalist human judges could reasonably evaluate the answers without specialized expertise.
Key Insights and Ideas
- The Verbosity Bias is Real: Every judge—human and AI—showed a preference for longer answers. As shown in the figure below, GPT-4 and Claude-1 were the most biased toward length, choosing the longer answer over 65% of the time. Distribution of decision choices based on answer length
- Style Masks Poor Substance: Crowd-sourced humans are particularly vulnerable to "hallucinations." They often rated long, factually incorrect answers higher than short, correct ones. Expert annotators were better at fact-checking but still struggled with the sheer volume of information.
- Humans are Indecisive: Human judges (especially experts) are far more likely to declare a "tie" than LLM judges. LLMs almost always pick a winner, which can lead to inflated Elo scores that don't reflect how close the competition actually was. Distribution of wins and ties across judges
- The "Halo Effect" in Evaluation: When judges give a single score, a model's good "Language" score often bleeds into its "Accuracy" score. By separating these dimensions in MERS, the authors found that GPT-4 became a much more rigorous fact-checker.
Key Results
The study's primary finding is summarized in the Elo ratings table, which shows how different "models" (error types) were ranked. Elo ratings for different error types
- LLMs as Judges: GPT-4 and Claude-1 are surprisingly good at ranking models by the severity of their factual errors, provided the answers are the same length. However, GPT-4's Elo for a "Correct + Short" answer (1096) was lower than for an answer with "Several Minor Factual Errors" (1206), proving that length can override truth.
- MERS Effectiveness: When using the Multi-Elo system, GPT-4’s ability to identify factual accuracy improved significantly. In the "Accuracy" dimension, it correctly ranked the "Correct" model much higher than those with factual errors, regardless of length.
- Crowd-Sourced Failure: Even with MERS, crowd-sourced workers showed little improvement in their ability to catch factual errors, suggesting that "the crowd" may not be a reliable source for high-stakes LLM evaluation.
Limitations and Future Work
The authors acknowledge that their study was limited to 40 questions, which may not represent the full complexity of LLM interactions. Furthermore, while MERS covers Accuracy, Helpfulness, and Language, there are other critical dimensions—like safety, bias, and creativity—that were not explored.
Future work is needed to understand why crowd-sourced workers remain "indecisive" even with multi-dimensional prompts and to find ways to scale expert-level human evaluation without the prohibitive cost. For now, the takeaway is clear: if you're evaluating an LLM, don't just ask "which is better?"—ask "which is more accurate?" and "which is better written?" separately.
Summary updated