DeltaScore: Fine-Grained Story Evaluation with Perturbations

Zhuohan Xie, Miao Li, Trevor Cohn, Jey Han Lau

TL;DR

DeltaScore is a novel evaluation metric for story generation that measures quality by "stress-testing" text with intentional perturbations. By calculating the drop in a story's likelihood under a language model after introducing errors (like typos or word shuffling), DeltaScore achieves state-of-the-art correlation with human judgments across fine-grained aspects like fluency, coherence, and logicality.

What Problem Does This Paper Address?

Evaluating creative writing is notoriously difficult for AI. Traditional metrics like BLEU or ROUGE, which rely on n-gram overlap with a reference text, fail to capture the "soul" of a story—its coherence, logical flow, or how interesting it is. While human evaluation remains the gold standard, it is expensive, slow, and hard to reproduce.

Recent model-based metrics (like BARTScore or GPTScore) have improved things by using the generative likelihood of a story as a proxy for quality. However, these often provide a single, blunt score and can be biased by simple factors like text length or word frequency. The authors of DeltaScore sought a way to provide fine-grained evaluation that actually understands the nuances of storytelling.

Methods

The core intuition behind DeltaScore is simple but powerful: High-quality stories are more sensitive to damage than low-quality ones.

Imagine a perfectly fluent sentence. If you add a typo, its probability under a language model drops significantly. Now imagine a garbled, low-quality sentence; adding one more typo barely changes its (already low) probability.

The intuition: High-quality stories (top) see a bigger likelihood drop when perturbed than low-quality ones (bottom).

The authors formalize this as: $$\text{DeltaScore}(s) = \log p(s|c) - \log p(s'|c)$$ Where $s$ is the original story, $s'$ is the perturbed version, and $c$ is the context (like a title).

They experimented with several perturbation strategies tailored to specific aspects:

  • Fluency: Adding typos or creating subject-verb disagreements.
  • Coherence: Shuffling words ("Jumble") or reordering sentences.
  • Relatedness: Removing words relevant to the title or replacing the entire story.
  • Logicality: Replacing words with antonyms or using ChatGPT to introduce commonsense violations.
  • Interestingness: Using ChatGPT to rewrite the story into a "blander" version.

Datasets

The researchers evaluated DeltaScore using two prominent story datasets:

  1. ROCStories (ROC): Short, 5-sentence stories focused on causal and temporal relations.
  2. WritingPrompts (WP): Longer, more complex fictional stories inspired by Reddit prompts.

They compared DeltaScore's performance against human ratings collected from both expert annotators (PhD students) and crowdworkers across five dimensions: fluency, coherence, relatedness, logicality, and interestingness.

Key Insights and Ideas

  • The "Jumble" Paradox: Surprisingly, the simplest perturbation—randomly shuffling words within a story—proved to be the most effective across almost all evaluation aspects. The authors hypothesize that "Jumble" acts as a natural normalization factor, canceling out the inherent biases that word frequency and sequence length usually introduce into likelihood calculations.
  • Sensitivity as a Proxy for Quality: The paper confirms that the "delta" (the difference) is a much stronger signal for quality than the raw likelihood score itself.
  • Model Size Matters (To a Point): While larger models like GPT-3.5 and OPT-66B generally provided better evaluation backbones, the benefits of scaling started to plateau, suggesting that once a model is "smart" enough to recognize basic story structures, further size increases yield diminishing returns for evaluation.

Key Results

DeltaScore consistently outperformed a wide array of existing metrics, including similarity-based ones (BLEU, BERTScore), discriminative ones (UNION, MANPLTS), and other generative metrics (BARTScore).

Figure 3: Absolute value of Story-level Kendall correlation (|τ|) between different metrics and crowdworker ratings. Higher bar indicates better performance. Red bars indicate DELTASCORE. Blue bars indicate similarity based metrics. Green bars indicate discriminative metrics. Purple bars indicate generative metrics. Gray bar indicates GPT3.5Eval.

As shown in Figure 3, DeltaScore (specifically the Jumble and Typo variants) achieved the highest Kendall correlation with human ratings. On the WritingPrompts dataset, DeltaScore reached correlation scores significantly higher than its closest competitors, proving its robustness for evaluating longer, more complex narratives.

Limitations and Future Work

While DeltaScore is a significant step forward, the authors acknowledge a few hurdles:

  • Interestingness is still a "Boss Level" problem: Even DeltaScore struggled to correlate strongly with human views on how "interesting" a story is, likely because this is the most subjective and context-dependent aspect of writing.
  • Perturbation Breadth: The study only looked at a specific set of perturbations. There may be more sophisticated ways to "damage" a story that reveal even deeper quality insights.
  • Evolving Benchmarks: The field moves fast; the authors note that very recent LLM-based evaluators like G-Eval were not included in this specific comparison.

Future work could involve applying DeltaScore to other tasks like machine translation or summarization, where the "sensitivity to perturbation" logic might also hold true.

Summary updated

Abstract

Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DELTASCORE, a novel methodology that employs perturbation techniques for the evaluation of nuanced story aspects. Our central proposition posits that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DELTASCORE with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DELTASCORE demonstrates remarkable performance, revealing a surprising finding that a specific perturbation proves highly effective in capturing multiple aspects.
BibTeX
@inproceedings{xie2023deltascore,
  author = {Zhuohan Xie and Miao Li and Trevor Cohn and Jey Han Lau},
  title = {{DeltaScore: Fine-Grained Story Evaluation with Perturbations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.353},
}
Figure