Abstract
Elizabeth Clark, Noah A. Smith. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
Evaluating open-ended, subjective tasks like story generation is a notorious challenge in Natural Language Processing (NLP). Standard automatic metrics (like BLEU or perplexity) often fail to correlate with human creativity, while traditional human evaluations typically rely on inconsistent Likert scales. These absolute ratings often fail to reflect how a model actually performs in a real-world, interactive writing context.
The authors address this by moving from absolute rating to comparative preference. They introduce CHOOSE YOUR OWN ADVENTURE (CYOA), a collaborative writing framework that evaluates story generation models through pairwise human-AI interaction. The goal is to determine not just if a model is "good," but if it is more useful than a baseline when a human is actively co-authoring a story.
The CYOA Framework
The core of the paper is the CYOA protocol, a turn-taking writing setup designed to capture nuanced preference and utility data.
As illustrated above, the process follows four distinct steps:
- Human Start: The writer pens the first sentence of a story.
- Paired Suggestions: Two different AI models each generate a suggestion for the next line.
- The Choice: The writer chooses one suggestion to continue their story.
- The Edit: The writer edits the chosen suggestion as much or as little as they like.
This setup generates three rich streams of data: Preference (which model was chosen), Utility (measured via Levenshtein edit distance and Jaccard similarity), and Stylistic Comparison (analyzing features like sentence length and vocabulary diversity).
Experimental Setup
The models were fine-tuned on the WritingPrompts dataset, a collection of over 300,000 story prompts from Reddit. The experiments were conducted using Amazon Mechanical Turk, involving over 200 human participants. The study primarily compared a Fusion model against GPT-2, and Nucleus sampling against Top-k sampling.
Key Results and Insights
The researchers found that a model's value often lies in its "utility" as a scaffold rather than its ability to mimic high literature.
GPT-2 vs. Fusion Model
GPT-2 was the clear winner, chosen 65.7% of the time. Writers kept significantly more of GPT-2's original text compared to the Fusion model, as reflected in the utility metrics.
As shown in the table, GPT-2 achieved a lower Edit Distance (ED) and higher Jaccard Similarity (JS) and User Story Edit Ratings (USER). GPT-2 tended to generate shorter, more concrete sentences, which writers found easier to integrate into their developing narratives, even if the model was more repetitive.
Decoding Strategies
The framework was sensitive enough to detect differences between decoding strategies using the same base model. While Top-k sampling was slightly preferred (53.4%), Nucleus sampling produced text that was stylistically closer to human writing in terms of vocabulary diversity (distinct-n metrics).
Conclusion
The CYOA framework demonstrates that "human-in-the-loop" signals—specifically the revisions humans make—can identify exactly where a model fails, whether it is losing the plot thread or being too generic. Future work includes expanding this framework to other creative domains and developing models that specifically optimize for "edit-friendliness" rather than just likelihood.
Summary updated