For decades, evaluating natural language generation (NLG) relied on a simple paradigm: compare the machine's output to a human-authored "gold standard" reference using token-matching heuristics like BLEU, ROUGE, or CIDEr. While this approach worked well for restrictive tasks like machine translation, it has become a major bottleneck for modern generative AI.
As large language models (LLMs) and multimodal systems tackle open-ended tasks—such as long-form text generation, creative storytelling, text simplification, and detailed image captioning—the space of valid outputs becomes effectively infinite. A single human reference cannot capture this diversity. As David Chan points out in Distribution Aware Metrics for Conditional Natural Language Generation, standard reference-based metrics penalize creative variety and reward models for playing it safe. Furthermore, Manik Bhandari notes in Re-evaluating Evaluation in Text Summarization that modern large-scale datasets rarely provide more than a single reference per document, severely limiting semantic evaluation.
Compounding this "gold-standard scarcity" is the sheer cost and logistical difficulty of gathering human references for specialized domains, real-time applications, or long-form documents (Dongfu Jiang et al.; Woosung Koh et al.). In fact, Marcella Cornia argues that generative models now routinely produce descriptions that exceed the detail and complexity of available human references.
To break this bottleneck, researchers have shifted toward reference-free evaluation. Instead of asking "How close is this output to what a human wrote?", reference-free metrics ask "How well does this output align with the source input, and does it make sense on its own?"
Direct Source Alignment
The most direct way to eliminate the reference text is to evaluate the generated output against the original source input. In tasks like text summarization or data-to-text generation, a high-quality output must faithfully preserve the core information of the source without omitting critical details or inventing new ones (hallucinations).
Early foundations in reference-free evaluation demonstrated that basic statistical properties could serve as a proxy for quality. For instance, Annie Louis et al. showed in Automatically Evaluating Content Selection in Summarization without Human Models that measuring the Jensen-Shannon divergence between the word distributions of a summary and its source document correlates remarkably well with human expert judgments. To capture deeper meaning beyond exact word matches, Elaheh ShafieiBavani et al. leveraged the compositionality of word and sense embeddings to compare summaries directly to their source texts.
Modern architectures scale this intuition by framing evaluation as an information retrieval or semantic correlation problem. David C. Uthus introduced RISE: Leveraging Retrieval Techniques for Summarization Evaluation, which uses a dual-encoder architecture to evaluate a summary based on how effectively a model can "retrieve" the original source document given only that summary. If the summary contains the essential, distinctive information of the source, the retrieval confidence will be high. Similarly, Qi Jia et al. developed a metric that combines semantic distribution correlation with compression ratios to judge summary quality directly against the source text, bypassing human references entirely.
Reframing Evaluation as Proxy Tasks
Another powerful intuition in reference-free evaluation is to transform the assessment into an auxiliary NLP task that a pre-trained model can solve. If a model can easily process or verify the generated text against the context, the text is likely high quality.
Cloze and Fill-in-the-Blank Tasks
Hao Zhou et al. developed CTRLEval, an unsupervised metric for controlled text generation. Instead of relying on references, CTRLEval formulates evaluation as a series of fill-in-the-blank tasks for a pre-trained language model. By measuring how predictably and coherently the generated text fits into various linguistic contexts, it assesses dimensions like coherence and consistency without requiring a ground-truth target.
Question Answering (QA) Verification
For multimodal tasks like image captioning, Hwanhee Lee et al. proposed QACE. Rather than matching words with a reference caption, QACE automatically generates questions based on the candidate caption and then attempts to answer them using the source image or text. If the answers match, the caption is factually grounded.
[Source Image/Text] ───► [QA Engine Answers Questions] ──┐
▼
[Candidate Caption] ───► [Generate Questions] ────► [Compare Answers] ───► Quality Score
Natural Language Inference (NLI)
For data-to-text generation, William Soto Martinez et al. utilized fine-tuned multilingual NLI models. By treating the source knowledge graph as a premise and the generated text as a hypothesis, they mathematically isolate precision (to detect hallucinations) and recall (to detect omissions), allowing for granular quality control across multiple languages.
Model-Based Judges and Explainable Feedback
With the advent of highly capable LLMs and Large Multimodal Models (LMMs), the field has increasingly adopted models as explicit, reference-free judges. Because these models possess strong zero-shot reasoning capabilities, they can follow complex evaluation rubrics directly.
To ensure these model judges are reliable and transparent, researchers have focused on generating natural language explanations alongside numerical scores. Dongfu Jiang et al. developed TIGERScore, an open-source metric that performs reference-free error analysis by diagnosing specific flaws in generated text. Similarly, Xinyu Hu et al. introduced Themis, which uses a rating-guided preference alignment strategy to provide interpretable, reference-free feedback. In the multimodal domain, Yebin Lee et al. introduced FLEUR, leveraging LMMs to evaluate image captions directly against the source image, providing both a score and a linguistic justification.
For highly complex or open-ended long-form text where a single score fails, Zhihan Guo et al. proposed ProxyReward. This framework breaks down a subjective prompt into a checklist of objective, boolean proxy questions, transforming a vague grading problem into a deterministic verification task.
[Complex Prompt] ───► [ProxyReward Checklist] ───► [Objective Verification] ───► Targeted Reward
By shifting the evaluation paradigm from surface-level similarity to structural, semantic, and logical verification, reference-free evaluation provides a scalable, robust framework that matches the open-ended nature of modern generative AI.
Key Foundational Work
GLEU: Automatic Evaluation of Sentence-Level Fluency (2007) — contributed 1 concept elements to this direction.
An Automatic Method for Summary Evaluation Using Multiple Evaluation Results by a Manual Method (2006) — contributed 1 concept elements to this direction.