All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text

Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, Noah A. Smith

TL;DR

As large language models (LLMs) like GPT-3 become incredibly fluent, it's harder than ever for humans to tell if text was written by a person or a machine. This paper reveals that untrained human evaluators are surprisingly bad at this, often focusing on superficial cues rather than true content. Even simple training methods only offer marginal improvements, suggesting we need to rethink how we evaluate generated text.

What Problem Does This Paper Address?

For years, "human evaluation" has been the gold standard for assessing the quality of text generated by Natural Language Generation (NLG) models. The idea is that if humans can't tell the difference between machine-generated text and human-written text, then the model is performing exceptionally well. However, as models like GPT-3 have become remarkably fluent, this assumption is being challenged. The core problem this paper addresses is: Can untrained human evaluators reliably distinguish between human-authored text and text generated by state-of-the-art LLMs, and if not, how can we improve this evaluation process? The authors highlight that current evaluation practices often rely on non-expert crowdworkers who may not have the time or expertise to deeply analyze generated content, leading to potentially misleading results.

Methods

The researchers conducted two main studies.

Study 1: Untrained Evaluators: They recruited Amazon Mechanical Turk (AMT) workers and presented them with text from three domains: stories, news articles, and recipes. Some texts were human-written, and others were generated by GPT-2 and GPT-3. Evaluators were asked to rate whether each text was "Definitely human-written," "Possibly human-written," "Possibly machine-generated," or "Definitely machine-generated." They also had to provide a brief explanation for their choice.

Study 2: Training Evaluators: Based on the findings from Study 1, the researchers explored three lightweight training methods to see if they could improve evaluators' ability to distinguish machine-generated text:

  1. Instruction-based training: Providing detailed instructions on what to look for (e.g., repetition, factuality) and what might be misleading (grammar, style).
  2. Example-based training: Presenting a few examples of human- and machine-written texts with explanations.
  3. Comparison-based training: Asking evaluators to directly compare two texts generated from the same prompt, one human and one machine, and guess which was which.

Datasets

The study used text from three domains:

  • Stories: Human stories were sourced from the Reddit WritingPrompts dataset. Machine-generated stories were created using GPT-2 and GPT-3, conditioned on human-written prompts.
  • News Articles: Human news articles were collected from various U.S. newspapers. Machine-generated articles were created using GPT-2 and GPT-3, conditioned on headlines and the first sentences of human articles.
  • Recipes: Human recipes were sourced from the RecipeNLG dataset. Machine-generated recipes were created using GPT-2 and GPT-3, conditioned on ingredient lists and titles of human recipes.

No new benchmarks were introduced; the focus was on evaluating existing NLG models and human evaluation methodologies.

Key Insights and Ideas

  • The "Gold Standard" is Tarnished: The most striking insight is that even with highly fluent models like GPT-3, untrained human evaluators perform no better than chance at distinguishing machine-generated text from human-written text across multiple domains.
  • Superficial Cues Dominate: Evaluators overwhelmingly focused on surface-level features like grammar, spelling, and style, rather than the deeper content or meaning of the text. This is problematic because modern LLMs excel at these surface-level aspects.
  • Misaligned Expectations: Evaluators often held outdated beliefs about what machines could or could not do, leading them to attribute text to humans based on perceived "human-like" qualities (like expressing emotions or personal descriptions) that LLMs are now capable of generating.
  • Training Offers Limited Gains: While some training methods (particularly example-based) showed marginal improvements in accuracy and a shift towards content-based reasoning, none significantly boosted performance across all domains. This suggests that simple interventions aren't enough to overcome the inherent difficulty.
  • Domain Matters (But Not Enough): While there were some domain-specific differences (e.g., stories were slightly easier to misattribute), the overall trend of poor performance held across stories, news, and recipes.

Key Results

  • Untrained Evaluators: Accuracy for distinguishing GPT-3 generated text from human text was around 49.9%, essentially random chance. For GPT-2, it was slightly better at 57.9%.
  • Training Impact: The "Examples" training method showed the most promise, improving accuracy slightly. However, this improvement was not statistically significant across all domains.
  • Focus Shift: After the "Examples" training, evaluators' explanations shifted from focusing on "form" (grammar, style) to "content," which is a more relevant dimension for evaluating advanced NLG models.
  • Low Agreement: Inter-annotator agreement (Krippendorff's alpha) was consistently low (around 0), indicating that even when evaluators did make a judgment, they often disagreed on their reasoning.

Limitations and Future Work

The authors acknowledge that their study focused on lightweight training methods suitable for crowdsourcing platforms like AMT. They suggest that more intensive, longer-term evaluations with expert annotators, or specialized third-party tools, might yield better results.

Future work should move away from purely intrinsic evaluations (e.g., "humanlikeness") and towards extrinsic evaluations that assess the usefulness of generated text in downstream tasks. They also recommend that researchers be more transparent about their evaluation methodologies, including details about training and instructions, to foster standardization and reproducibility in NLG evaluation. The paper also hints at the need to debunk common misconceptions about AI capabilities among evaluators.

Summary updated

BibTeX
@inproceedings{clark2021all,
  author = {Elizabeth Clark and Tal August and Sofia Serrano and Nikita Haduong and Suchin Gururangan and Noah A. Smith},
  title = {{All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=9ttV8W4GZTA},
}
Figure