Automated Proxies for Human Text Evaluation

This direction is merged into another direction: manual evaluation bottleneck. Its last version stays readable and citable here.

Version 1 of 122 papers22 conceptsfrozen 2026-10-04

The central challenge in Natural Language Generation (NLG) is not just making models that speak, but knowing when they are speaking well. While human judgment remains the "gold standard" for assessing text quality, it is notoriously slow, expensive, and difficult to scale. As Cristina Garbacea and Samuel Carton note in Judge the Judges, the rapid iteration of modern AI requires "automated judges" that can simulate human intuition without the human overhead.

Research in this field has evolved from simple word-matching to sophisticated semantic verification, attempting to bridge the gap between what a computer can count and what a human can understand.

The Limits of Surface Matching

Early efforts focused on lexical overlap. Chin-Yew Lin introduced ROUGE in Looking for a Few Good Metrics, which measures how many n-grams (sequences of words) a machine-generated summary shares with a human reference. This worked well for early extractive systems, but as models became more fluent, these metrics began to fail.

The problem is that two sentences can share zero words but mean exactly the same thing. Conversely, a model can achieve a high ROUGE score by repeating keywords while failing to convey any actual meaning. Andrea Sottana and colleagues argue in Evaluation Metrics in the Era of GPT-4 that classic metrics are increasingly unreliable for state-of-the-art models, which often produce outputs that humans prefer over the "gold standard" references themselves.

Decomposing Text into Atomic Facts

To move beyond surface words, researchers began breaking text into "atomic units" of information. Simone Teufel and Hans van Halteren pioneered Factoid Analysis in Evaluating Information Content by Factoid Analysis. Instead of comparing sentences, they compared "factoids"—the smallest possible units of information. If a human summary says "The cat sat on the mat," the factoids might be (1) there is a cat, (2) there is a mat, and (3) the cat is on the mat.

This approach is highly stable but manually intensive. To automate this, Mohit Bansal and Shiyue Zhang 0001 developed the Lite-Pyramid family of metrics in Finding a Balanced Degree of Automation for Summary Evaluation. By using Natural Language Inference (NLI) to check if a generated summary "entails" the facts found in a reference, they created a way to trade off manual effort for evaluation accuracy.

Models as Judges

The current frontier involves using Large Language Models (LLMs) themselves as the evaluators. This shift relies on the intuition that a model might be better at recognizing quality than generating it.

  1. Fine-Grained Error Analysis: In Toward Human-Like Evaluation for Natural Language Generation with Error Analysis, Liang Ding and Derek F. Wong introduced BARTScore++. This metric mimics an expert judge by attempting to "detect and correct" errors in a sentence. The more effort the model requires to fix a sentence, the lower its score.
  2. Pairwise Ranking: For subjective tasks like countering hate speech, Irune Zubiaga and Rodrigo Agerri suggest a tournament-style approach in A LLM-based Ranking Method. Instead of giving a raw score, an LLM judge compares two responses side-by-side, which more closely mirrors how humans make comparative judgments.
  3. The Open-Book Test: Perhaps the most intuitive recent development is ProxyQA by Haochen Tan and Zhijiang Guo. In ProxyQA: An Alternative Framework, they argue that a good summary should be able to answer specific questions about the source material. By generating "proxy questions" from the source and seeing if the summary contains the answers, they turn evaluation into a verifiable test of information density.

Why This Works

These modern metrics work because they shift the focus from form (how it looks) to function (what it does). By breaking text into verifiable units—whether through factoids, error detection, or question-answering—researchers are creating automated proxies that don't just count words, but verify the presence and accuracy of information, much like a human reader would.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_proxies_for_human_text_evaluat_416cd42b,
  title = {Automated Proxies for Human Text Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-proxies-for-human-text-evaluation/rdr_416cd42b574d449a8c0972478b84d953},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_proxies_for_human_text_evaluat_416cd42b_v1,
  title = {Automated Proxies for Human Text Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-proxies-for-human-text-evaluation/rdr_416cd42b574d449a8c0972478b84d953/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}