Style Over Substance: Evaluation Biases for Large Language Models

Minghao Wu, Alham Fikri Aji

Abstract

As large language models (LLMs) continue to advance, accurately and comprehensively evaluating their performance becomes increasingly challenging. Human evaluations are conventionally considered the gold standard in natural language generation, but recent advancements incorporate state-of-the-art LLMs as proxies for human judges in evaluation processes. However, the extent to which humans and LLMs are capable evaluators remains uncertain. This study investigates the behavior of crowd-sourced and expert annotators, as well as LLMs, when comparing outputs from different models. To achieve this, we curate a dataset of intentionally flawed machine-generated answers. Our findings reveal a concerning bias in the evaluation process, as answers with factual errors are rated more favorably than answers that are too short or contained grammatical errors. To address this issue, we propose independently evaluating machine-generated text across multiple dimensions, rather than merging all the evaluation aspects into a single score. We instantiate this idea with the Elo rating system, resulting in the Multi-Elo Rating System. Empirical results from our study reveal that this proposed approach significantly enhances the quality of LLM-based evaluations, particularly in terms of factual accuracy. However, there is no significant improvement in crowd-sourced-based evaluations, indicating the need for further investigation and refinement.

Introduction

The rapid evolution of Large Language Models (LLMs) has outpaced traditional static benchmarks. Consequently, the industry has shifted toward pairwise human evaluation, often aggregated using the Elo rating system. While this "human-in-the-loop" approach is widely regarded as the gold standard, it relies on the assumption that judges—whether human or AI—are objective and capable of identifying subtle flaws. This paper investigates whether evaluators are being swayed by stylistic heuristics, such as length and linguistic fluency, at the expense of factual substance.

Methodology: The "Flawed" Dataset

To isolate specific biases, we developed a controlled dataset of 480 responses across 40 general-purpose questions. Instead of using organic model outputs, we intentionally generated 12 variants for each question to simulate specific failure modes:

  • Factual Integrity: Variants included "Correct" answers, as well as those with "Minor" or "Major" factual hallucinations.
  • Linguistic Proficiency: Variants simulated "Advanced" or "Intermediate" English learners by introducing spelling and grammatical errors.
  • Length Constraints: Each variant was produced in a "Long" (~100 words) and "Short" (~50 words) version.

We then utilized four distinct judge types—Crowd-sourced workers (AMT), Expert annotators, GPT-4, and Claude-1—to rank these responses using standard pairwise comparison frameworks.

Analysis of Evaluation Biases

The Verbosity Bias

A primary finding of this study is the pervasive "verbosity bias" across all judge categories. Evaluators consistently favored longer responses, even when they contained significant factual errors. As illustrated in Figure 3, both human and LLM judges showed a clear preference for longer answers, with GPT-4 and Claude-1 being particularly susceptible to this heuristic.

Figure 3: Win rates for longer vs. shorter answers across different judge types, demonstrating a clear verbosity bias.

Style vs. Substance

Perhaps more concerning is the tendency for judges to prioritize linguistic style over factual accuracy. Our results indicate that grammatical and spelling errors are penalized far more severely than factual hallucinations.

Figure 1: Elo ratings for different model variants. Note how linguistic errors (Intermediate Learner) often result in lower ratings than major factual errors for certain judges.

As shown in Figure 1, crowd-sourced workers were particularly poor at identifying factual errors; their Elo ratings for models with major factual lies were nearly indistinguishable from their ratings for correct models. Furthermore, GPT-4 penalized an "Intermediate Learner" (771 Elo) more harshly than a model with "Several Major Factual Errors" (861 Elo), suggesting that a confident, well-spoken liar is more likely to win an evaluation than a truthful but unpolished assistant.

The Multi-Elo Rating System (MERS)

To mitigate these biases, we propose the Multi-Elo Rating System (MERS). Rather than a single holistic score, MERS requires judges to evaluate outputs across three independent dimensions: Accuracy, Helpfulness, and Language.

Our empirical results demonstrate that by decomposing the evaluation task, we can significantly improve the reliability of LLM-based judges. When GPT-4 was specifically tasked with evaluating the "Accuracy" dimension, it was able to correctly rank factual models, effectively separating substance from style. However, this improvement was less pronounced in crowd-sourced human evaluations, suggesting that non-expert humans may lack the necessary domain knowledge or diligence to perform fine-grained factual verification, regardless of the framework used.

Conclusion

This study highlights a critical vulnerability in current LLM evaluation pipelines. If we continue to optimize models based on holistic human or AI preferences, we risk incentivizing "hallucination with style." Adopting multi-dimensional frameworks like MERS is a necessary step toward building AI systems that are not just convincing, but consistently accurate.

Summary updated

Abstract

As large language models (LLMs) continue to advance, accurately and comprehensively evaluating their performance becomes increasingly challenging. Human evaluations are conventionally considered the gold standard in natural language generation, but recent advancements incorporate state-of-the-art LLMs as proxies for human judges in evaluation processes. However, the extent to which humans and LLMs are capable evaluators remains uncertain. This study investigates the behavior of crowd-sourced and expert annotators, as well as LLMs, when comparing outputs from different models. To achieve this, we curate a dataset of intentionally flawed machine-generated answers. Our findings reveal a concerning bias in the evaluation process, as answers with factual errors are rated more favorably than answers that are too short or contained grammatical errors. To address this issue, we propose independently evaluating machine-generated text across multiple dimensions, rather than merging all the evaluation aspects into a single score. We instantiate this idea with the Elo rating system, resulting in the Multi-Elo Rating System. Empirical results from our study reveal that this proposed approach significantly enhances the quality of LLM-based evaluations, particularly in terms of factual accuracy. However, there is no significant improvement in crowd-sourced-based evaluations, indicating the need for further investigation and refinement.
BibTeX
@inproceedings{wu2025style,
  author = {Minghao Wu and Alham Fikri Aji},
  title = {{Style Over Substance: Evaluation Biases for Large Language Models}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure