TL;DR
DeltaScore is a novel evaluation metric for story generation that measures quality by "stress-testing" text with intentional perturbations. By calculating the drop in a story's likelihood under a language model after introducing errors (like typos or word shuffling), DeltaScore achieves state-of-the-art correlation with human judgments across fine-grained aspects like fluency, coherence, and logicality.
What Problem Does This Paper Address?
Evaluating creative writing is notoriously difficult for AI. Traditional metrics like BLEU or ROUGE, which rely on n-gram overlap with a reference text, fail to capture the "soul" of a story—its coherence, logical flow, or how interesting it is. While human evaluation remains the gold standard, it is expensive, slow, and hard to reproduce.
Recent model-based metrics (like BARTScore or GPTScore) have improved things by using the generative likelihood of a story as a proxy for quality. However, these often provide a single, blunt score and can be biased by simple factors like text length or word frequency. The authors of DeltaScore sought a way to provide fine-grained evaluation that actually understands the nuances of storytelling.
Methods
The core intuition behind DeltaScore is simple but powerful: High-quality stories are more sensitive to damage than low-quality ones.
Imagine a perfectly fluent sentence. If you add a typo, its probability under a language model drops significantly. Now imagine a garbled, low-quality sentence; adding one more typo barely changes its (already low) probability.
The authors formalize this as: $$\text{DeltaScore}(s) = \log p(s|c) - \log p(s'|c)$$ Where $s$ is the original story, $s'$ is the perturbed version, and $c$ is the context (like a title).
They experimented with several perturbation strategies tailored to specific aspects:
- Fluency: Adding typos or creating subject-verb disagreements.
- Coherence: Shuffling words ("Jumble") or reordering sentences.
- Relatedness: Removing words relevant to the title or replacing the entire story.
- Logicality: Replacing words with antonyms or using ChatGPT to introduce commonsense violations.
- Interestingness: Using ChatGPT to rewrite the story into a "blander" version.
Datasets
The researchers evaluated DeltaScore using two prominent story datasets:
- ROCStories (ROC): Short, 5-sentence stories focused on causal and temporal relations.
- WritingPrompts (WP): Longer, more complex fictional stories inspired by Reddit prompts.
They compared DeltaScore's performance against human ratings collected from both expert annotators (PhD students) and crowdworkers across five dimensions: fluency, coherence, relatedness, logicality, and interestingness.
Key Insights and Ideas
- The "Jumble" Paradox: Surprisingly, the simplest perturbation—randomly shuffling words within a story—proved to be the most effective across almost all evaluation aspects. The authors hypothesize that "Jumble" acts as a natural normalization factor, canceling out the inherent biases that word frequency and sequence length usually introduce into likelihood calculations.
- Sensitivity as a Proxy for Quality: The paper confirms that the "delta" (the difference) is a much stronger signal for quality than the raw likelihood score itself.
- Model Size Matters (To a Point): While larger models like GPT-3.5 and OPT-66B generally provided better evaluation backbones, the benefits of scaling started to plateau, suggesting that once a model is "smart" enough to recognize basic story structures, further size increases yield diminishing returns for evaluation.
Key Results
DeltaScore consistently outperformed a wide array of existing metrics, including similarity-based ones (BLEU, BERTScore), discriminative ones (UNION, MANPLTS), and other generative metrics (BARTScore).
As shown in Figure 3, DeltaScore (specifically the Jumble and Typo variants) achieved the highest Kendall correlation with human ratings. On the WritingPrompts dataset, DeltaScore reached correlation scores significantly higher than its closest competitors, proving its robustness for evaluating longer, more complex narratives.
Limitations and Future Work
While DeltaScore is a significant step forward, the authors acknowledge a few hurdles:
- Interestingness is still a "Boss Level" problem: Even DeltaScore struggled to correlate strongly with human views on how "interesting" a story is, likely because this is the most subjective and context-dependent aspect of writing.
- Perturbation Breadth: The study only looked at a specific set of perturbations. There may be more sophisticated ways to "damage" a story that reveal even deeper quality insights.
- Evolving Benchmarks: The field moves fast; the authors note that very recent LLM-based evaluators like G-Eval were not included in this specific comparison.
Future work could involve applying DeltaScore to other tasks like machine translation or summarization, where the "sensitivity to perturbation" logic might also hold true.
Summary updated