Introduction
Evaluating the quality of natural language generation (NLG) systems is a persistent challenge in AI research. Traditional metrics like BLEU or ROUGE often fail to capture semantic coherence, while human evaluations using Likert scales are expensive and often lack granularity. This paper introduces RoFT (Real or Fake Text), a gamified platform designed to crowdsource the detection of boundaries where human-written text transitions into machine-generated content.
The RoFT Platform
RoFT employs a novel boundary detection task to evaluate text quality. Unlike standard preference tests, RoFT presents users with a passage one sentence at a time. The initial sentences are human-written, but at a specific, hidden point, a machine model takes over the generation. Users are tasked with identifying this exact transition point.
To ensure high-quality data, the platform is gamified: users receive maximum points for identifying the exact boundary, with points decreasing for late guesses. No points are awarded for premature guesses. This setup encourages users to be both attentive and decisive. Following a guess, users provide a brief natural language explanation for their decision, creating a rich dataset of "tells" or errors that characterize current NLG models.
Experimental Setup
The researchers tested two primary domains to evaluate the platform:
- News: Using the New York Times Annotated Corpus with completions by the GROVER model.
- Stories: Using Reddit Writing Prompts with completions by GPT-2 XL.
Both models utilized Nucleus Sampling, with the $p$ parameter varied to explore the relationship between sampling diversity and human detectability.
Results and Analysis
A pilot study involving 200 participants yielded several key insights into how humans perceive machine-generated text.
Detection Accuracy
Humans find it remarkably difficult to identify the exact moment a machine takes over. Only 15.8% of annotations correctly identified the precise boundary sentence. On average, users guessed 1.989 sentences after the true boundary, suggesting that modern models can maintain local fluency and context well enough to fool readers for several sentences.
As shown in the histogram, the distribution of guesses is heavily skewed toward positive values (late guesses). This indicates that while models eventually fail—often due to logical inconsistencies or repetitive phrasing—they are initially quite convincing. Interestingly, the study found that users did not significantly improve their detection skills over a session of 10 annotations, suggesting the task remains challenging even with experience.
Qualitative "Tells"
The free-form feedback provided by users highlighted common failure modes in NLG. Frequent reasons for flagging text as "machine-generated" included logical entailment failures, unnatural repetition, and sudden formatting shifts. These explanations provide a goldmine for researchers looking to diagnose specific weaknesses in their models.
Conclusion
RoFT provides a scalable, cost-effective method for evaluating NLG systems by focusing on long-term coherence rather than just local fluency. The results demonstrate that while current models are adept at short-term mimicry, they struggle to maintain the logical structure required to fool humans over longer passages. Future work will leverage the RoFT platform to conduct larger studies across diverse datasets and decoding strategies, ultimately aiming to create more robust automated evaluation metrics.
Summary updated