SNaC: Coherence Error Detection for Narrative Summarization

Tanya Goyal, Junyi Jessy Li, Greg Durrett

TL;DR

SNaC (Summary Narrative Coherence) is a new framework and dataset designed to detect and localize fine-grained coherence errors in long-form narrative summaries. By moving beyond holistic "quality" scores to specific span-level annotations, the authors demonstrate that current state-of-the-art models frequently fail at narrative flow—and that their new classifier can identify these failures far better than standard metrics like ROUGE.

What Problem Does This Paper Address?

As Large Language Models (LLMs) scale, they have become remarkably good at generating fluent, on-topic sentences. However, when tasked with summarizing long documents like books or movie scripts, the "big picture" often falls apart. A summary might be grammatically perfect but narratively nonsensical—introducing characters out of nowhere, jumping abruptly between scenes, or referencing events that were never mentioned.

Current evaluation methods are ill-equipped for this. Automatic metrics like ROUGE focus on word overlap, which doesn't capture flow. Human evaluation typically relies on Likert scales (e.g., "rate coherence from 1-5"), which are highly subjective and result in low inter-annotator agreement for long texts. The authors argue that we need a way to pinpoint where and why a narrative breaks down.

Methods

The authors introduce a structured approach to coherence evaluation:

  1. Error Taxonomy: They developed a schema of seven error types, split into two categories:
  2. Coherence Errors: New character without introduction (CharE), missing reference to an event/object (RefE), abrupt scene transitions (SceneE), and logical inconsistencies (InconE).
  3. Language & Fluency Errors: Unnecessary repetition (RepE), ungrammatical/nonsensical text (GramE), and unclear coreference (CorefE).
  4. Annotation Protocol: Instead of asking workers to rate a 500-word summary at once, they decomposed the task. Annotators evaluated small segments (2-4 sentences) while having access to the preceding context. They highlighted specific spans and assigned error labels.
  5. Automated Detection: The authors fine-tuned a T5-Large model on their collected annotations to automatically detect and localize these errors. They compared this "SNaC-trained" model against unsupervised methods (like GPT-2 perplexity) and models trained on synthetic data (artificially corrupted text).
Figure 2: SNAC’s error schema. Given context, i.e., the generated summary until that point, annotators identify error spans in the current summary segment. We define two high-level error categories: (1) Coherence Errors that directly affect narrative understanding, and (2) Language Errors that measure other aspects, like grammar.

Datasets

The study utilized several key sources to build the SNaC Dataset:

  • Book Summaries: Generated by OpenAI’s GPT-3 (175B and 6B versions) using the BookSum dataset.
  • Movie Summaries: Generated using a BART-based model (Summ^N) on the TRIPOD dataset.
  • The SNaC Dataset itself: Contains 9.6k span-level annotations across 160 summaries (6.6k sentences). This includes both expert annotations and crowdsourced data from trained Mechanical Turk workers.

Key Insights and Ideas

  • Holistic Scores are "Gameable": The authors found that even if you randomly shuffle the sentences of a summary or repeat 50% of them, ROUGE scores barely budge. This highlights a dangerous blind spot in how we currently measure summarization progress.
  • Granularity Improves Agreement: While humans struggle to agree on a single 1-5 coherence score for a long summary, they show much higher agreement when asked to identify specific spans where a new character appears without introduction.
  • Narrative Summarization is Unique: The errors found in summaries (like missing context) are fundamentally different from errors in open-ended story generation (which are dominated by repetition). This justifies the need for a summarization-specific taxonomy.
  • Synthetic Data is a Poor Substitute: Models trained on "fake" errors (like random sentence swapping) performed poorly on real-world model failures. Training on actual human-identified errors is essential for building reliable coherence classifiers.
Figure 3: An example of expert annotations for a BOOK-175B summary (we only show coherence errors). The number of annotators who identified each span is denoted by v; for simplicity, we omit v = 1 errors. We see that annotators often identify overlapping error spans; this fine-grained picture of coherence cannot be achieved by a summary-level score.

Key Results

  • Model Performance: The fine-tuned T5 model achieved strong results in localizing errors, particularly for CharE (F1 of 0.80) and SceneE (F1 of 0.40-0.53).
  • Human vs. Machine: Interestingly, the trained model often achieved higher recall than individual human annotators. Because the model was trained on the aggregated knowledge of multiple annotators, it became more sensitive to subtle errors that a single human might overlook.
  • Error Prevalence: The analysis revealed that state-of-the-art models are still struggling significantly; each generated summary contained an average of ~30 coherence errors of varying severity.

Limitations and Future Work

  • Domain & Language: The study is limited to English-language narratives (books and movies). The taxonomy might need adjustments for other domains like government reports or technical documentation.
  • Subjectivity of "Missing Info": The RefE (Missing Reference) category remains the hardest for both humans and models to agree on, as "what context is necessary" can be subjective.
  • Future Applications: The authors suggest that SNaC could be used not just for evaluation, but for post-hoc correction. A model could detect a coherence error (e.g., a missing character intro) and then prompt a secondary model to "fix" that specific span by adding the necessary context.

For those interested in exploring the data or models, the authors have released their work at github.com/tagoyal/snac.

Summary updated

Abstract

Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. When a long summary must be produced to appropriately cover the facets of that text, that summary needs to present a coherent narrative to be understandable by a reader, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators. Furthermore, we show that the collected annotations allow us to train a strong classifier for automatically localizing coherence errors in generated summaries as well as benchmarking past work in coherence modeling. Finally, our SNaC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and post-hoc summary correction.
BibTeX
@inproceedings{goyal2022snac,
  author = {Tanya Goyal and Junyi Jessy Li and Greg Durrett},
  title = {{SNaC: Coherence Error Detection for Narrative Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=OMvDcklcnM},
}
Figure