SNaC: Coherence Error Detection for Narrative Summarization

Tanya Goyal, Junyi Jessy Li, Greg Durrett

Introduction

While Large Language Models (LLMs) have achieved remarkable fluency in short-form text generation, long-form narrative summarization remains a significant challenge. Modern models can summarize entire books or movies, but the resulting text often feels like a "list of events stapled together" rather than a cohesive story. This phenomenon, termed narrative incoherence, manifests as characters appearing without introduction, abrupt scene transitions, or internal factual contradictions.

Standard evaluation metrics such as ROUGE or BERTScore are largely "blind" to these discourse-level issues. Because they focus on word overlap or semantic similarity, they fail to penalize structural disarray. Furthermore, holistic human ratings (e.g., Likert scales) suffer from low inter-annotator agreement when applied to long summaries, as different readers prioritize different types of errors. To address these gaps, we introduce SNaC (Summary Narrative Coherence), a framework and dataset designed for fine-grained, span-level detection of coherence errors.

The SNaC Framework

The core of SNaC is a shift from holistic scoring to localized, categorized error detection. We developed a taxonomy of seven error types divided into two categories:

  1. Coherence Errors:
    • CharE: Introduction of a new character without context.
  2. RefE: Reference to an event or object as if previously mentioned.
  3. SceneE: Abrupt or jarring transitions between narrative scenes.
  4. InconE: Factual inconsistencies or contradictions within the summary.
  5. Language & Fluency Errors: Including unnecessary repetition (RepE), ungrammatical or nonsensical phrasing (GramE), and unclear coreference (CorefE).

Segmented Annotation Workflow

To improve human agreement, SNaC utilizes a "step-by-step" annotation process. Instead of evaluating a full summary at once, annotators are presented with small segments (2–4 sentences) alongside the preceding context. This ensures that errors are identified relative to the established narrative flow.

The SNaC annotation workflow presents segments in context to improve error detection accuracy.

This methodology significantly improved inter-annotator agreement, with Krippendorff’s $\alpha$ rising from a dismal 0.19 on holistic scales to 0.69 for specific categories like character errors.

Experimental Results

We applied the SNaC framework to summaries generated by various models, including GPT-3 (BOOK-175B, BOOK-6B) and BART-based models (MOVIE-BART). The resulting SNaC dataset contains 6.6k sentences and approximately 9.6k span-level annotations.

Quantifying Incoherence

Our analysis reveals that even state-of-the-art models like GPT-3 175B struggle with global planning. While they maintain high local fluency, they frequently fail at "connectedness." As shown in the distribution of errors, character introduction (CharE) and scene transitions (SceneE) are among the most prevalent issues across all model types.

Distribution of coherence and language errors across different summarization models and datasets.

Key findings from our experiments include:

  • Metric Gameability: We demonstrated that a summary consisting merely of a list of named entities can achieve ROUGE scores comparable to human-written summaries, highlighting the inadequacy of overlap-based metrics.
  • Synthetic Data Limitations: Models trained on "fake" errors (e.g., shuffled sentences) do not generalize well to the subtle narrative breaks produced by LLMs. In contrast, a T5 model fine-tuned on the SNaC dataset achieved high recall for real-world errors, such as Character (0.90) and Scene Transitions (0.82).
  • Human vs. Machine: The SNaC-trained model often acted as a highly effective "red flag" system, identifying potential narrative breaks with higher recall than individual human annotators, though humans maintained higher precision.

Conclusion

SNaC provides a rigorous framework for diagnosing why AI summaries "lose the plot." By moving toward fine-grained, context-aware error detection, we can better evaluate the narrative capabilities of LLMs. Future work will focus on using SNaC to build "post-hoc" correction systems that can automatically identify and fix coherence gaps, ultimately leading to AI-generated summaries that are as narratively sound as they are fluent.

Summary updated

Abstract

Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. When a long summary must be produced to appropriately cover the facets of that text, that summary needs to present a coherent narrative to be understandable by a reader, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators. Furthermore, we show that the collected annotations allow us to train a strong classifier for automatically localizing coherence errors in generated summaries as well as benchmarking past work in coherence modeling. Finally, our SNaC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and post-hoc summary correction.
BibTeX
@inproceedings{goyal2022snac,
  author = {Tanya Goyal and Junyi Jessy Li and Greg Durrett},
  title = {{SNaC: Coherence Error Detection for Narrative Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=OMvDcklcnM},
}
Figure