When we evaluate natural language generation (NLG) models, we often treat language as a deterministic, one-to-one mapping. We provide a prompt, collect a single human-written response, and use string-matching or semantic similarity metrics to score the model's output against that lone reference.
This approach is a trap. Language is inherently open-ended, characterized by what Mario Giulianelli et al. describe as "production variability" or aleatoric uncertainty (What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability). For any given semantic input, human speakers naturally select from a vast array of valid lexical choices, syntactic structures, and stylistic voices (Underspecifying and Predicting Voice for Surface Realisation Ranking).
As Kathleen McKeown et al. argued during the early scaling of text generation systems, evaluating against a single human reference introduces a severe negative bias because it ignores this natural diversity (Lessons Learned from Large Scale Evaluation of Systems that Produce Text: Nightmares and Pleasant Surprises). When an evaluation framework relies on only one "gold standard," perfectly fluent and accurate model outputs are routinely penalized as false negatives simply because they do not match the specific phrasing chosen by a single annotator. To build reliable metrics, the evaluation target must expand from a single point to a multi-reference space.
The Cost of Rigidity: Underestimation Across Tasks
The bias of single-reference evaluation is not a theoretical nuance; it fundamentally distorts our understanding of model capabilities across nearly every text generation domain.
Grammatical Error Correction (GEC)
An ungrammatical sentence rarely has a single unique fix. As Kengo Hotate et al. point out, GEC is an underspecified task where multiple distinct corrections can be equally valid and fluent (Generating Diverse Corrections with Local Beam Search for Grammatical Error Correction). In the CoNLL-2013 shared task, Hwee Tou Ng et al. demonstrated that traditional string-matching metrics unfairly penalize systems that provide correct alternative edits, and that incorporating these alternatives significantly boosts measured performance (The CoNLL-2013 Shared Task on Grammatical Error Correction).
Subsequent work by Alla Rozovskaya et al. confirmed that accounting for multiple valid corrections prevents the severe underestimation of system utility (The Illinois-Columbia System in the CoNLL-2014 Shared Task). In Chinese GEC, Bo Zhang et al. observed the same pattern: single-reference frameworks mask true model capabilities by penalizing legitimate linguistic variations (MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction). Remarkably, when Christopher Bryant et al. expanded an evaluation set to include 10 human expert corrections per sentence, they revealed that top GEC systems were performing at over 73% of human proficiency—a far more optimistic and accurate assessment than previous single-reference metrics allowed (How Far are We from Fully Automatic High Quality Grammatical Error Correction?).
Sentiment and Reading Comprehension
The same underestimation occurs in structured understanding tasks. In Aspect-Based Sentiment Analysis, models are often penalized for predicting semantically accurate terms that do not exactly match a single ground-truth span. Soyoung Yang et al. showed that expanding reference sets to include diverse but valid expressions yields a much higher correlation with human judgment (Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation).
Similarly, in machine reading comprehension and automated short-answer grading, datasets like MS MARCO (Tri Nguyen et al., MS MARCO: A Human Generated MAchine Reading COmprehension Dataset) and grading systems developed by Ifeyinwa Okoye et al. (CU : Computational Assessment of Short Free Text Answers - A Tool for Evaluating Students' Understanding) demonstrate that grading accuracy scales up when student or model responses are compared against multiple valid reference answers. Providing a broader target space allows similarity-based classifiers to capture the diverse ways a concept can be correctly expressed.
Mechanisms of Multi-Reference Matching
To evaluate a generated text against a set of multiple references rather than a single string, metrics must alter their underlying aggregation logic. The two primary paradigms for doing this are max-pooling and distributional alignment.
Max-Pooling (Closest-Match Evaluation)
The most common mechanism for multi-reference evaluation is to calculate the similarity score between the generated hypothesis and each available reference independently, and then select the maximum score.
$$\text{Score}(h) = \max_{r \in R} \text{Similarity}(h, r)$$
As Naeha Sharif et al. demonstrate with learned composite metrics for image captioning, this max-pooling approach ensures that a candidate is judged based on its closest valid human-authored counterpart (LCEval: Learned Composite Metric for Caption Evaluation).
In speech recognition and dialectal speech evaluation, where orthographic standards vary wildly, Ahmed Ali 0002 et al. use this closest-match logic to prevent artificial inflation of Word Error Rates (Towards Variability Resistant Dialectal Speech Evaluation). For discrete tasks like phonetic transliteration (Haizhou Li et al., Whitepaper of NEWS 2010 Shared Task on Transliteration Generation) or fill-in-the-blanks video understanding (Santiago Castro et al., FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework), this mechanism treats any match against a pre-verified set of synonyms or taxonomic variants as a successful generation, completely removing the penalty for valid stylistic deviations.
Distributional Alignment
While max-pooling identifies whether a model's single output matches one valid human path, it fails to evaluate whether a model can capture the broader distribution of human language. To address this, David Chan et al. introduced Distribution Aware Metrics (Distribution Aware Metrics for Conditional Natural Language Generation).
Instead of performing one-to-one comparisons, these metrics compare a set of generated hypotheses against a set of human references. By measuring the distance between these two distributions, researchers can evaluate both the quality and the creative variety of the generated text. This prevents models from "playing it safe" by generating a generic, averaged response that scores moderately well against everything but represents natural human expression poorly.
Expanding the Reference Space
The primary bottleneck of multi-reference evaluation is data density: human annotations are expensive and difficult to scale. The field has split its efforts between high-density crowdsourcing and automated reference expansion.
High-Density Crowdsourcing
For highly variable tasks like text simplification, researchers have curated specialized datasets to capture human rewriting strategies. The ASSET dataset (Fernando Alva-Manchego et al., ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations) and the MCTS dataset (Ruining Chong et al., MCTS: A Multi-Reference Chinese Text Simplification Dataset) collect multiple independent human simplifications for each source sentence. This allows metrics to reward diverse but equally correct operations—such as sentence splitting, lexical substitution, and structural compression—rather than forcing models down a single path.
Similarly, in long-document question-focused summarization, Alex Wang et al. built SQuALITY to provide multiple independent human summaries, capturing the inherent variability in how professional writers synthesize information (SQuALITY: Building a Long-Document Summarization Dataset the Hard Way). In reasoning tasks, Naoya Inoue et al. collect multiple human-authored step-by-step derivations to validate that a model reached a conclusion via a legitimate reasoning chain (R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason).
Automated Reference Expansion
To bypass the cost of human curation, recent work uses generative models to synthesize alternative references. Shinhyeok Oh et al. demonstrated that explicitly modeling data-space diversity through textual paraphrases significantly improves the alignment between automated scores and human quality assessments in question generation (Evaluation of Question Generation Needs More References).
Building on this intuition, Tianyi Tang et al. developed Div-Ref, a framework that leverages Large Language Models to automatically generate a diverse pool of high-quality reference sentences for tasks like translation and summarization (Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References).
The core insight of the Div-Ref framework provides a fitting conclusion to the multi-reference philosophy: the perceived flaws of standard metrics like BLEU or ROUGE are often not caused by the metric algorithms themselves, but by the poverty of the reference sets they are forced to use. By expanding a single ground-truth reference into a diverse semantic space, we allow simple, robust metrics to finally align with human judgment.
Key Foundational Work
Lessons Learned from Large Scale Evaluation of Systems that Produce Text: Nightmares and Pleasant Surprises (2006) — contributed 4 concept elements to this direction.
Bleu: a Method for Automatic Evaluation of Machine Translation (2002) — contributed 1 concept elements to this direction.