When evaluating natural language generation (NLG) systems, a troubling paradox frequently emerges: automated metrics regularly score machine-generated text higher than human-authored text, even when human experts overwhelmingly prefer the human versions.
This phenomenon is not a sign that machines have surpassed human capability. Rather, it exposes a fundamental flaw in how our evaluation systems operate. Across generations of metrics—from traditional n-gram overlap scores to modern large language model (LLM) judges—there is a systematic bias toward surface-level statistical patterns, word frequencies, and model-like fluency at the expense of genuine linguistic diversity, creativity, and nuance.
The Frequency Bias: Favoring the Commonplace
The root of this misalignment lies in how metrics calculate text quality. Early automatic evaluation metrics were built on the assumption that high-quality text should mirror the statistical properties of a standard corpus. However, as shown by Aoife Cahill in Correlating Human and Automatic Evaluation of a German Surface Realiser, these metrics exhibit a strong frequency bias. They favor generated text that closely aligns with common word frequencies, failing to capture the naturalness or structural correctness that human judges prioritize.
Similarly, Anya Belz et al. demonstrated in Comparing Automatic and Human Evaluation of NLG Systems that metrics like BLEU and NIST disproportionately reward NLG systems that rely heavily on frequent words. By prioritizing statistical prevalence, these metrics penalize systems that employ less common phrasing, even when those choices are made to improve clarity or introduce stylistic variation that human readers value.
The Reference Penalty and the Diversity Gap
When metrics rely on direct comparison to a single human-written "gold standard" reference, they introduce a severe penalty against linguistic diversity. Human language is inherently varied; there are dozens of valid ways to express the same idea. Yet, as Jungo Kasai et al. argue in Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand, n-gram overlap metrics prioritize surface-level fluency over semantic accuracy.
Because machine generation models are trained to output highly probable, average sequences, they often mimic the specific phrasing or statistical profile of a reference more closely than a creative human writer would. Consequently, diverse human-written text receives lower scores than machine outputs simply because it deviates from the narrow vocabulary of the provided reference.
This "diversity gap" is further explored by Shuqian Sheng et al. in Is Reference Necessary in the Evaluation of NLG Systems? When and Where?. The authors note that reference-based metrics consistently penalize highly fluent and valid text if it uses alternative vocabulary. This dynamic creates an artificial ceiling: as machine text approaches human-level excellence, traditional metrics become highly unreliable because they mistake creative divergence for poor quality.
The Echo Chamber of Reference-Free and LLM Judges
To bypass the limitations of fixed references, the NLP community has increasingly turned to reference-free metrics and LLM-as-a-judge frameworks. However, shifting away from references does not eliminate the underlying bias; instead, it shifts the bias toward the internal statistical patterns of the evaluator models themselves.
In On the Limitations of Reference-Free Evaluations of Generated Text, Daniel Deutsch et al. reveal that reference-free metrics frequently assign higher scores to machine-generated text than to human-written text. This occurs because these metrics evaluate quality based on how well a text conforms to the probability distributions of large language models. They prioritize uniform, model-like fluency over the complexity, nuance, and creative irregularities characteristic of natural human language.
This creates a self-reinforcing echo chamber. As Anna Bavaresco et al. demonstrate in LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks, automated judges experience a notable drop in evaluation performance when judging machine-generated text compared to human-authored text. The authors highlight a systematic bias: automated evaluators favor their own linguistic patterns, making them unreliable objective benchmarks for new generative models.
Exploitation and the Challenge for System Optimization
Because automatic metrics track statistical properties rather than true semantic understanding, they are highly vulnerable to exploitation. Ozan Caglayan et al. illustrate this vulnerability in Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale. They show that popular metrics like BLEU and METEOR are insensitive to rare word errors while over-penalizing common word substitutions, allowing simple adversarial changes to easily fool the metrics into overrating low-quality synthetic outputs.
This misalignment introduces severe complications when metrics are used directly to optimize text generation. For instance, in Minimum Bayes Risk (MBR) decoding—a technique that selects a generation by maximizing its expected quality under a given metric—the discrepancy between metric scores and human preferences creates a persistent bottleneck. As analyzed by Hidetaka Kamigaito et al. in Theoretical Aspects of Bias and Diversity in Minimum Bayes Risk Decoding, this "bias" component is incredibly difficult to reduce mathematically because the metrics themselves are fundamentally misaligned with human judgment. Consequently, optimizing for metric scores often leads to a loss of linguistic diversity and an inflation of superficial fluency, masking factual inaccuracies and stylistic stagnation.
Key Foundational Work
Correlating Human and Automatic Evaluation of a German Surface Realiser (2009) — contributed 2 concept elements to this direction.
Comparing Automatic and Human Evaluation of NLG Systems (2006) — contributed 1 concept elements to this direction.