What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025

Jing Yang, Nils Feldhus, Salar Mohtaj, Leonhard Hennig, Qianli Wang, Eleni Metheniti, Sherzod Hakimov, Charlott Jakob, Veronika Solopova, Konrad Rieck, David Schlangen, Sebastian Möller, Vera Schmitt

TL;DR

As Natural Language Generation (NLG) expands, researchers are increasingly turning to Large Language Models to evaluate other LLMs—a paradigm known as "LLM-as-a-Judge" (LaaJ), which officially surpassed human evaluation in 2025. However, a massive meta-analysis of over 3,300 papers reveals a dangerous "validation gap": LaaJ adoption is skyrocketing without proper human validation, and its alignment with human judgment completely collapses on fine-grained criteria like fluency and coherence.

What Problem Does This Paper Address?

Evaluating how well an AI generates text has become a massive bottleneck in AI research. As NLG models have shifted from rigid tasks like translation to highly open-ended applications (like creative writing or conversational agents), traditional evaluation methods have broken down. Human evaluation is slow, expensive, and difficult to scale, which has driven researchers to rapidly adopt LLMs as automated judges.

However, the field has adopted these automated judges without a clear understanding of their reliability. This paper provides the first large-scale, systematic analysis of NLG evaluation trends from 2020 to 2025, tackling three systemic challenges:

  1. Metric Inertia: The stubborn persistence of outdated, word-matching metrics (like BLEU and ROUGE) for open-ended text.
  2. The Metric-Criteria Mapping Problem: Using general-purpose metrics as a blanket proxy for "quality" without defining what specific dimension of text is being measured.
  3. The Validation Gap: The explosive growth of LLM-as-a-Judge methods without corresponding human validation.

Methods

To analyze an entire scientific field's output, the authors built a scalable, multi-LLM information extraction pipeline. They collected 14,171 papers from four major NLP conferences (ACL, EMNLP, NAACL, and INLG) spanning 2020 to 2025.

Using three prominent open-source LLMs (DeepSeek-R1, GPT-OSS-120B, and Qwen3-235B), the pipeline extracted structured metadata by answering four binary questions regarding the presence of NLG tasks, automatic metrics, LaaJ, and human evaluation. The results were harmonized using majority voting and verified by a separate LLM (DeepSeek V3.1). Raw evaluation criteria were then mapped to a standardized taxonomy (QCET). To ensure the pipeline's accuracy, a team of 11 human NLP researchers manually annotated a subset of papers, demonstrating exceptionally high agreement with the automated pipeline.

Datasets

Instead of evaluating models on standard benchmarks, this study performs a meta-analysis on the NLP literature itself. The primary dataset consists of the 14,171 papers from the ACL Anthology (2020–2025). After filtering out non-NLG tasks (like code generation or text classification), the authors focused their analysis on a core subset of 3,334 papers representing the top 30 NLG tasks.

Key Insights and Ideas

  • The Great Substitution: LLM-as-a-Judge is actively cannibalizing human evaluation. While human evaluation has steadily declined, LaaJ has surged from under 1% before 2023 to 46% in 2025.
  • We Are Flying Blind: Despite LaaJ's explosive growth, fewer than 8% of all analyzed NLG papers (and only 16% of papers using LaaJ in 2025) explicitly validate their LLM judge's scores against human judgments.
  • Superficial Alignment: LLM judges are reasonably good at grading "Overall Quality" but struggle significantly with fine-grained analysis. When it comes to strict linguistic nuances like fluency or coherence, the alignment between LLM judges and humans completely degrades.
  • Decoupled Evaluation: Researchers routinely select metrics based on popularity or tradition rather than a deliberate methodological match to the criteria they claim to measure.

Key Results

  • The Overtaking of Human Eval: In 2025, LaaJ adoption hit 46%, officially overtaking human evaluation, which dropped to 43%. Meanwhile, basic automatic metrics remain completely ubiquitous at ~95%.
Figure 1: Evaluation method adoption across the 3,334 Top-30 NLG-task papers (2020–2025). Automatic met- rics remain ubiquitous (∼95%). LaaJ adoption surged from less than 1% before 2023 to 46% in 2025, surpass- ing human evaluation, which declined from 51% to 43%. Yet only 16% of papers in 2025 explicitly validate LaaJ against human evaluation, indicating a validation gap.
  • Independent Metric Selection: A within-task co-occurrence analysis revealed that metrics and criteria are largely decoupled. Significant pairings only concentrate in generic, high-frequency quadrants, meaning task-distinctive criteria are rarely paired with task-distinctive metrics.
Figure 5: Within-task co-occurrence LRs for metric–criterion pairs. Cells are split into upper-left orange (LaaJ subset) and lower-right blue (human subset) triangles, with color intensity reflecting association strength. Outlined cells indicate significance. Axes: Rows (metrics) and columns (criteria) show the top-5 items by frequency (black) and the top-5 by task-LR (red), separated by dotted lines. Significant cells concentrate in the frequency×frequency quadrant (light yellow areas) across DG and MT, while the distinctiveness×distinctiveness quadrant is empty in all four tasks and QA shows no significant pairings under either method.
  • The Fluency Collapse: Quantitative analysis of LaaJ-human alignment showed that while automated judges achieve moderately high correlations for holistic quality, median correlations for specific dimensions like fluency and coherence drop near or below 0.4 with massive variance.
Figure 6: Distribution of LaaJ vs. human evaluation scores by metric and criterion. Individual observations are shown as gray dots. n denotes the number of studies included. Values are based on human annotated 70 papers.

Limitations and Future Work

The authors openly acknowledge several limitations in their extraction pipeline:

  • Paper-Level vs. Task-Level Extraction: For papers dealing with multiple NLG tasks, metadata was extracted at the paper level rather than being mapped to each individual task, which may introduce minor numerical deviations.
  • PDF Parsing Losses: The pipeline suffered from under-extraction when identifying fine-grained criterion comparisons. This was primarily caused by PDF-to-text conversion tools losing tabular content where correlation scores are typically reported.
  • Context-Light Mapping: When mapping raw criteria to the QCET taxonomy, the LLM only evaluated the criterion name string without surrounding paper context, which occasionally limited mapping precision.

Future work will focus on improving LLM information extraction from scientific literature, particularly in parsing complex tables and capturing richer contextual definitions.

Summary updated

Abstract

As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.
BibTeX
@misc{yang2026what,
  author = {Jing Yang and Nils Feldhus and Salar Mohtaj and Leonhard Hennig and Qianli Wang and Eleni Metheniti and Sherzod Hakimov and Charlott Jakob and Veronika Solopova and Konrad Rieck and David Schlangen and Sebastian Möller and Vera Schmitt},
  title = {{What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025}},
  year = {2026},
  eprint = {2601.07648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.07648},
}
Figure