The central challenge in Natural Language Generation (NLG) is not just making models that speak, but knowing when they are speaking well. While human judgment remains the "gold standard" for assessing text quality, it is notoriously slow, expensive, and difficult to scale. As Cristina Garbacea and Samuel Carton note in Judge the Judges, the rapid iteration of modern AI requires "automated judges" that can simulate human intuition without the human overhead.
Research in this field has evolved from simple word-matching to sophisticated semantic verification, attempting to bridge the gap between what a computer can count and what a human can understand.
The Limits of Surface Matching
Early efforts focused on lexical overlap. Chin-Yew Lin introduced ROUGE in Looking for a Few Good Metrics, which measures how many n-grams (sequences of words) a machine-generated summary shares with a human reference. This worked well for early extractive systems, but as models became more fluent, these metrics began to fail.
The problem is that two sentences can share zero words but mean exactly the same thing. Conversely, a model can achieve a high ROUGE score by repeating keywords while failing to convey any actual meaning. Andrea Sottana and colleagues argue in Evaluation Metrics in the Era of GPT-4 that classic metrics are increasingly unreliable for state-of-the-art models, which often produce outputs that humans prefer over the "gold standard" references themselves.
Decomposing Text into Atomic Facts
To move beyond surface words, researchers began breaking text into "atomic units" of information. Simone Teufel and Hans van Halteren pioneered Factoid Analysis in Evaluating Information Content by Factoid Analysis. Instead of comparing sentences, they compared "factoids"—the smallest possible units of information. If a human summary says "The cat sat on the mat," the factoids might be (1) there is a cat, (2) there is a mat, and (3) the cat is on the mat.
This approach is highly stable but manually intensive. To automate this, Mohit Bansal and Shiyue Zhang 0001 developed the Lite-Pyramid family of metrics in Finding a Balanced Degree of Automation for Summary Evaluation. By using Natural Language Inference (NLI) to check if a generated summary "entails" the facts found in a reference, they created a way to trade off manual effort for evaluation accuracy.
Models as Judges
The current frontier involves using Large Language Models (LLMs) themselves as the evaluators. This shift relies on the intuition that a model might be better at recognizing quality than generating it.
- Fine-Grained Error Analysis: In Toward Human-Like Evaluation for Natural Language Generation with Error Analysis, Liang Ding and Derek F. Wong introduced BARTScore++. This metric mimics an expert judge by attempting to "detect and correct" errors in a sentence. The more effort the model requires to fix a sentence, the lower its score.
- Pairwise Ranking: For subjective tasks like countering hate speech, Irune Zubiaga and Rodrigo Agerri suggest a tournament-style approach in A LLM-based Ranking Method. Instead of giving a raw score, an LLM judge compares two responses side-by-side, which more closely mirrors how humans make comparative judgments.
- The Open-Book Test: Perhaps the most intuitive recent development is ProxyQA by Haochen Tan and Zhijiang Guo. In ProxyQA: An Alternative Framework, they argue that a good summary should be able to answer specific questions about the source material. By generating "proxy questions" from the source and seeing if the summary contains the answers, they turn evaluation into a verifiable test of information density.
Why This Works
These modern metrics work because they shift the focus from form (how it looks) to function (what it does). By breaking text into verifiable units—whether through factoids, error detection, or question-answering—researchers are creating automated proxies that don't just count words, but verify the presence and accuracy of information, much like a human reader would.