Abstract
Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. When a long summary must be produced to appropriately cover the facets of that text, that summary needs to present a coherent narrative to be understandable by a reader, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators. Furthermore, we show that the collected annotations allow us to train a strong classifier for automatically localizing coherence errors in generated summaries as well as benchmarking past work in coherence modeling. Finally, our SNaC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and post-hoc summary correction.
Introduction: The Narrative Gap
While Large Language Models (LLMs) have demonstrated remarkable proficiency in summarizing short-form content like news articles, long-form narrative summarization—encompassing books and movie screenplays—remains a significant challenge. Current models often produce summaries that, while factually grounded, lack narrative flow. These outputs frequently read like a "stapled together" list of events rather than a coherent story.
The primary obstacle to improving these models is evaluation. Standard automatic metrics such as ROUGE or BERTScore focus on word overlap and semantic similarity but remain "blind" to discourse structure. Furthermore, human evaluation via holistic Likert scales (e.g., rating a summary 1–5) has proven notoriously unreliable for long texts, often yielding abysmal inter-annotator agreement because "coherence" is too vague a concept to rate holistically.
The SNaC Framework
To address these limitations, we introduce SNaC (Summary Narrative Coherence), a framework rooted in fine-grained, span-level error detection. Rather than asking for a general quality score, SNaC requires annotators to identify specific narrative disruptions.
A Taxonomy of Narrative Errors
The framework defines seven specific error types categorized into two primary groups:
- Coherence Errors:
- Character Errors (CharE): Introducing a character without proper context or introduction.
- Reference Errors (RefE): Missing references to previous events or entities necessary for understanding.
- Scene Errors (SceneE): Abrupt or illogical transitions between scenes.
- Inconsistency Errors (InconE): Factual contradictions within the narrative flow.
- Language Errors: These include repetition, ungrammatical phrasing, and unclear coreferences.
Annotation Protocol
To improve reliability, SNaC employs a segment-by-segment annotation protocol. Annotators evaluate summaries in small chunks (2–4 sentences) rather than reading the entire text at once. This method forces focus on local transitions and specific spans of text, leading to significantly higher agreement compared to holistic ratings.
Experimental Setup and Datasets
The SNaC dataset was constructed using summaries generated by state-of-the-art models, including GPT-3 (175B and 6B versions) and BART. The source materials were drawn from BookSum (long-form book chapters) and Movie-BART (screenplays).
The resulting dataset contains approximately 9.6k span-level annotations across 160 summaries, covering 6.6k sentences. This represents the first major characterization of the specific coherence "hallucinations" prevalent in modern LLM-generated narratives.
Key Insights
The Failure of Holistic Metrics
Analysis confirms that Likert scales are insufficient for long-form narrative evaluation. Human agreement on a 1–5 scale for book summaries was found to be a mere $\alpha = 0.19$. By shifting to specific error types, SNaC provides a much more stable and actionable metric for researchers.
The "Event List" Trap and Metric Gaming
Models frequently fall into the "event list" trap, where they identify important plot points but fail to provide the connective tissue required for a coherent narrative. Alarmingly, standard metrics like ROUGE are easily "gamed." Experiments showed that randomly shuffling sentences or repeating 50% of the text resulted in negligible changes to ROUGE scores, proving that current leaderboards may reward "bag-of-words" accuracy while ignoring actual readability and logic.
The Role of Synthetic Data
The study found that training models on "synthetic" errors—such as randomly shuffled sentences—is a poor substitute for real-world data. Such models fail to detect the subtle, logical errors made by actual LLMs. Effective detection requires training on human-labeled mistakes to understand the nuances of narrative flow.
Results: Automated Detection and Localization
We fine-tuned T5-Large models on the SNaC dataset to evaluate the feasibility of automated coherence "redlining."
- Detection Performance: The model achieved high performance in specific categories, notably an F1 of 0.80 for character introduction errors and 0.53 for scene transitions.
- Model vs. Human: While humans are precise, they often miss subtle errors. The trained T5 model demonstrated higher recall, consistently flagging potential issues that individual human annotators overlooked.
- Localization: The models successfully pinpointed the exact spans of text causing incoherence, with overlap scores ranging from 0.87 to 1.0 for top error categories.
Conclusion and Future Work
SNaC provides a robust framework for diagnosing the narrative failures of modern summarization models. While currently focused on English-language narratives, the framework lays the groundwork for several future directions, including post-hoc correction (automatically rewriting incoherent spans) and severity weighting to distinguish between minor scene jumps and major character introduction failures. Ultimately, SNaC can serve as a training signal to encourage models to generate more coherent discourse structures from the outset.
Summary updated