Abstract
Despite advances in Natural Language Generation (NLG), evaluation remains challenging. Although various new metrics and LLM-as-a-judge (LaaJ) methods are proposed, human judgment persists as the gold standard. To systematically review how NLG evaluation has evolved, we employ an automatic information extraction scheme to gather key information from NLG papers, focusing on different evaluation methods (metrics, LaaJ and human evaluation). With extracted metadata from 14,171 papers across four major conferences (ACL, EMNLP, NAACL, and INLG) over the past six years, we reveal several critical findings: (1) Task Divergence: While Dialogue Generation demonstrates a rapid shift toward LaaJ (>40% in 2025), Machine Translation remains locked into n-gram metrics, and Question Answering exhibits a substantial decline in the proportion of studies conducting human evaluation. (2) Metric Inertia: Despite the development of semantic metrics, general-purpose metrics (e.g., BLEU, ROUGE) continue to be widely used across tasks without empirical justification, often lacking the discriminative power to distinguish between specific quality criteria. (3) Human-LaaJ Divergence: Our association analysis challenges the assumption that LLMs act as mere proxies for humans; LaaJ and human evaluations prioritize very different signals, and explicit validation is scarce (<8% of papers comparing the two), with only moderate to low correlation. Based on these observations, we derive practical recommendations to improve the rigor of future NLG evaluation.
Introduction
As Large Language Models (LLMs) have moved beyond simple translation to complex tasks like creative writing and technical reporting, our methods for measuring their success have struggled to keep pace. The field currently relies on three main pillars: traditional mathematical metrics (e.g., BLEU, ROUGE), the "gold standard" of human evaluation, and the emerging paradigm of LLM-as-a-Judge (LaaJ). However, the community is largely "flying blind" regarding how these methods align. This study maps these trends across 14,171 research papers to bring order to the evaluation court and address the growing issue of "metric inertia."
Methodology: A Meta-Analysis at Scale
To analyze the vast volume of research published between 2020 and 2025, we developed an automated information extraction pipeline. This pipeline utilized three state-of-the-art models (DeepSeek-R1, GPT-OSS-120B, and Qwen3) to extract metadata from papers in the ACL Anthology. A harmonization step followed, using majority voting and a fourth LLM to normalize terminology (e.g., consolidating variations of "BLEU"). The pipeline's accuracy was validated by 11 NLP researchers who manually annotated a subset of papers, confirming high agreement. The study also tracked frequently used benchmarks, identifying GSM8K (math reasoning), CNN/DailyMail (summarization), and WMT14 (translation) as dominant datasets in the literature.
The Great Task Divergence
Our analysis reveals a "fractured landscape" where different sub-fields of NLP are evolving at vastly different speeds. Dialogue Generation is leading the transition toward automated judgment, with over 40% of papers adopting LaaJ by 2025. In contrast, Machine Translation remains largely tethered to traditional n-gram metrics like BLEU.
A concerning trend is observed in Question Answering (QA); while the volume of QA research is growing rapidly, the proportion of studies incorporating human evaluation has seen a substantial decline. This shift suggests a growing reliance on automated proxies without a corresponding increase in rigorous validation.
Metric Inertia and the Validation Gap
Despite the rise of "semantic" metrics like BERTScore that understand meaning, researchers are stubbornly clinging to n-gram overlap metrics like BLEU and ROUGE, even for tasks where they are known to be poor indicators of quality. Furthermore, a critical "validation gap" exists: fewer than 8% of papers that use an LLM as a judge actually compare that judge's scores to human judgments to ensure they correlate.
LaaJ vs. Human Judgment: A Divergent Reality
A central finding of this study is that LLMs are not mere proxies for human judges. Association analysis shows that humans and LLMs prioritize fundamentally different quality signals. Humans tend to focus on Naturalness and Appropriateness, whereas LLM judges are more strongly associated with criteria like Helpfulness and Harmlessness.
When validation does occur, the correlation for "Overall Quality" is moderate ($\mu = 0.47–0.71$), but it drops significantly for finer linguistic traits such as Fluency ($\mu = 0.35$). This suggests that while LaaJ may be useful for high-level technical adherence, it remains a poor substitute for the nuanced perception of human language.
Conclusion and Future Directions
The current state of NLG evaluation is characterized by rapid but unvalidated adoption of automated judges. To improve rigor, we recommend a stratified evaluation protocol: use LaaJ for technical adherence (formatting, instruction following) but reserve human evaluation for high-level dimensions like creativity, safety, and naturalness, where AI judges still diverge from human perception. Moving forward, explicit validation of automated judges against human benchmarks must become a standard practice in the field.
Summary updated