All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text

Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, Noah A. Smith

Abstract

Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts' ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators' accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.

The Problem with the 'Gold Standard'

In Natural Language Generation (NLG), human judgment has long been the ultimate metric for quality. The assumption is simple: if a human believes a text is human-written, the model has succeeded. However, as large language models (LLMs) like GPT-3 reach near-human levels of fluency, this assumption creates a "Turing-style" trap.

This research investigates a critical gap: the reliability of non-expert evaluators, such as those recruited via crowdsourcing platforms like Amazon Mechanical Turk (AMT). If these evaluators are easily fooled or rely on flawed reasoning, our primary method for measuring AI progress becomes fundamentally unreliable. The study explores whether untrained humans can spot the difference across various domains and whether lightweight training can improve their detection skills.

Study Methodology

The researchers conducted a large-scale study using AMT workers to evaluate text across three domains: stories (Reddit WritingPrompts), news articles (Newspaper3k), and recipes (RecipeNLG). Evaluators rated text on a 4-point scale from "Definitely human-written" to "Definitely machine-generated."

To understand the reasoning behind these choices, the authors collected qualitative explanations. They also tested three training interventions:

  1. Instructions: Specific tips on what to look for (e.g., repetition) and what to ignore (e.g., bad grammar).
  2. Examples: Showing three annotated examples of human and machine text before the task.
  3. Comparison: Showing side-by-side pairs of human and machine-generated text for the same prompt.

The Illusion of Fluency and Contradictory Logic

A key finding was the "Fluency Illusion." Evaluators frequently mistook GPT-3's high grammatical fluency for human intent, citing "emotional depth" or "creative phrasing" in machine-generated text. Conversely, they often focused on surface-level features like grammar and spelling. Because modern LLMs excel at these "form" elements, evaluators missed deeper "content" errors like logical inconsistencies or factual hallucinations.

Evaluators often used contradictory logic to justify their labels, as seen in these thought bubbles comparing human and machine text.

Qualitative analysis revealed that evaluators often used the same evidence to reach opposite conclusions. One might label a phrase "too creative for a bot," while another called the same phrase "too weird for a human." Additionally, a strong "human bias" was observed: two-thirds of all guesses were "human," even though the actual split was 50/50.

Quantitative Findings

The results show a clear decline in detection accuracy as models improve. For GPT-2, evaluators were correct 58% of the time. However, for GPT-3, accuracy dropped to 49.9%—essentially a coin flip. Inter-annotator agreement (Krippendorff’s alpha) was near zero, indicating that evaluators were not only often wrong but also inconsistent with one another.

Performance metrics for human evaluators across different domains and models. Accuracy for GPT-3 was near chance level.

While training interventions like providing examples helped slightly—raising accuracy to 55%—the improvement was inconsistent across domains. Training did successfully shift evaluators' attention from surface-level "form" to deeper "content," but this shift did not reliably translate into correct judgments.

Conclusion and Recommendations

The study concludes that "human-likeness" is no longer a sufficient standalone metric for NLG. The authors recommend that researchers move away from simple detection tasks and toward "extrinsic" evaluations that measure whether generated text is useful for specific tasks. Furthermore, they suggest providing evaluators with better training and more specific, content-focused criteria to move beyond the surface-level traps of modern LLM fluency.

Summary updated

Abstract

Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts' ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators' accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.
BibTeX
@inproceedings{clark2021all,
  author = {Elizabeth Clark and Tal August and Sofia Serrano and Nikita Haduong and Suchin Gururangan and Noah A. Smith},
  title = {{All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=9ttV8W4GZTA},
}
Figure