Abstract
Evaluating creative writing is inherently subjective and challenging for automated systems. While Large Language Models (LLMs) are frequently used as evaluators, they often fall into a "fluency trap," where high-quality prose in justifications masks poor alignment with actual scores. We present CoKe (Chain-of-Keyword), a method that introduces a content bottleneck by forcing models to generate salient keywords before producing rationales and scores. By sampling multiple keyword chains to simulate diverse human perspectives, CoKe achieves human-level performance on fine-grained story metrics, significantly outperforming much larger models like GPT-4.
Introduction
Standard evaluation methods for creative writing often rely on free-text explanations, such as Chain-of-Thought (CoT) prompting. However, these rationales frequently contain "artifacts"—sentiment-laden phrases like "I feel" or "It is wonderful"—that bias the model toward specific scores regardless of the story's actual content. Furthermore, human judgment is naturally diverse; a single "point-of-view" score often fails to represent the consensus of a target population.
To address these issues, we introduce CoKe. By bottlenecking the reasoning process through keywords, we ensure the model grounds its judgment in specific story concepts. Additionally, by employing a Monte Carlo simulation over multiple keyword sequences, CoKe captures the inherent subjectivity of human crowds.
Methodology
The CoKe framework consists of a specialized pipeline with two primary components: the Rationalizer and the Scorer.
The Rationalizer (T5-3B)
The Rationalizer is fine-tuned to generate a Chain-of-Keywords followed by a full rationale. During training, keywords are extracted from human justifications using TextRank. At inference, the model samples multiple distinct keyword sequences (e.g., "mature woman," "white sand," "night sky") to explore various interpretative "angles" of the story.
The Scorer (DeBERTa-V3)
The Scorer predicts a Likert score (1–5) based on the story, the specific aspect under evaluation (e.g., "Character Shaping"), and the generated keywords. To simulate a crowd's opinion, CoKe uses a Monte Carlo simulation, sampling $N$ (up to 100) different keyword sequences, scoring each, and averaging the results. This marginalization mimics the process of aggregating opinions from multiple human annotators.
Experimental Setup
The study utilizes the StoryER dataset, which consists of human-written stories from the Reddit r/WritingPrompts community. The dataset includes multi-annotator ratings across 10 fine-grained aspects, such as Ending, Scene Description, and Novelty/Idea, accompanied by free-text justifications. The dataset was partitioned into approximately 18k training, 4.5k development, and 5.6k test instances.
Results and Discussion
CoKe demonstrates significant improvements over existing baselines across several dimensions.
Human-Level Performance
CoKe achieved a Pearson correlation ($ ho$) of 0.3142, surpassing the average human-to-human correlation of 0.3037 on the StoryER dataset. This indicates that the model's aggregated judgments are as consistent with human consensus as individual humans are with each other.
Comparison with Large Language Models
Despite its smaller size (3B parameters), CoKe significantly outperformed GPT-4, which achieved a correlation of only ~0.11. It also reduced the Mean Squared Error (MSE) to 0.0811, a substantial improvement over GPT-3.5's 0.5172. These results suggest that targeted architectural design and fine-tuning can be more effective for subjective tasks than raw scale.
The Importance of Diversity
As shown in the results, the model's performance increases steadily with the number of sampled keyword sequences ($N$). This confirms that simulating a "crowd" through diverse rationales is essential for accurate subjective evaluation. Furthermore, the keyword bottleneck effectively mitigates "artifact bias," preventing the model from relying on superficial linguistic cues.
Conclusion
CoKe offers a robust, customizable approach to story evaluation by leveraging a keyword-based reasoning bottleneck and crowd simulation. While currently focused on creative writing, the principles of content bottlenecking and rationale marginalization hold promise for other subjective NLP tasks. Future work will explore the scalability of this approach to larger reasoning architectures and its generalizability to broader domains such as essay grading.
Summary updated