CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization

Brihi Joshi, Sriram Venkatapathy, Mohit Bansal, Nanyun Peng, Haw-Shiuan Chang

Abstract

Evaluating creative writing is inherently subjective and challenging for automated systems. While Large Language Models (LLMs) are frequently used as evaluators, they often fall into a "fluency trap," where high-quality prose in justifications masks poor alignment with actual scores. We present CoKe (Chain-of-Keyword), a method that introduces a content bottleneck by forcing models to generate salient keywords before producing rationales and scores. By sampling multiple keyword chains to simulate diverse human perspectives, CoKe achieves human-level performance on fine-grained story metrics, significantly outperforming much larger models like GPT-4.

Introduction

Standard evaluation methods for creative writing often rely on free-text explanations, such as Chain-of-Thought (CoT) prompting. However, these rationales frequently contain "artifacts"—sentiment-laden phrases like "I feel" or "It is wonderful"—that bias the model toward specific scores regardless of the story's actual content. Furthermore, human judgment is naturally diverse; a single "point-of-view" score often fails to represent the consensus of a target population.

To address these issues, we introduce CoKe. By bottlenecking the reasoning process through keywords, we ensure the model grounds its judgment in specific story concepts. Additionally, by employing a Monte Carlo simulation over multiple keyword sequences, CoKe captures the inherent subjectivity of human crowds.

Methodology

The CoKe framework consists of a specialized pipeline with two primary components: the Rationalizer and the Scorer.

The CoKe pipeline for training and inference, illustrating the keyword extraction and explanation generation process.

The Rationalizer (T5-3B)

The Rationalizer is fine-tuned to generate a Chain-of-Keywords followed by a full rationale. During training, keywords are extracted from human justifications using TextRank. At inference, the model samples multiple distinct keyword sequences (e.g., "mature woman," "white sand," "night sky") to explore various interpretative "angles" of the story.

The Scorer (DeBERTa-V3)

The Scorer predicts a Likert score (1–5) based on the story, the specific aspect under evaluation (e.g., "Character Shaping"), and the generated keywords. To simulate a crowd's opinion, CoKe uses a Monte Carlo simulation, sampling $N$ (up to 100) different keyword sequences, scoring each, and averaging the results. This marginalization mimics the process of aggregating opinions from multiple human annotators.

Experimental Setup

The study utilizes the StoryER dataset, which consists of human-written stories from the Reddit r/WritingPrompts community. The dataset includes multi-annotator ratings across 10 fine-grained aspects, such as Ending, Scene Description, and Novelty/Idea, accompanied by free-text justifications. The dataset was partitioned into approximately 18k training, 4.5k development, and 5.6k test instances.

Results and Discussion

CoKe demonstrates significant improvements over existing baselines across several dimensions.

Human-Level Performance

CoKe achieved a Pearson correlation ($ ho$) of 0.3142, surpassing the average human-to-human correlation of 0.3037 on the StoryER dataset. This indicates that the model's aggregated judgments are as consistent with human consensus as individual humans are with each other.

Pearson correlation performance of CoKe compared to baselines and human performance as the number of samples (N) increases.

Comparison with Large Language Models

Despite its smaller size (3B parameters), CoKe significantly outperformed GPT-4, which achieved a correlation of only ~0.11. It also reduced the Mean Squared Error (MSE) to 0.0811, a substantial improvement over GPT-3.5's 0.5172. These results suggest that targeted architectural design and fine-tuning can be more effective for subjective tasks than raw scale.

The Importance of Diversity

As shown in the results, the model's performance increases steadily with the number of sampled keyword sequences ($N$). This confirms that simulating a "crowd" through diverse rationales is essential for accurate subjective evaluation. Furthermore, the keyword bottleneck effectively mitigates "artifact bias," preventing the model from relying on superficial linguistic cues.

Conclusion

CoKe offers a robust, customizable approach to story evaluation by leveraging a keyword-based reasoning bottleneck and crowd simulation. While currently focused on creative writing, the principles of content bottlenecking and rationale marginalization hold promise for other subjective NLP tasks. Future work will explore the scalability of this approach to larger reasoning architectures and its generalizability to broader domains such as essay grading.

Summary updated

BibTeX
@inproceedings{joshi2025coke,
  author = {Brihi Joshi and Sriram Venkatapathy and Mohit Bansal and Nanyun Peng and Haw-Shiuan Chang},
  title = {{CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure