Introduction
While Large Language Models (LLMs) have achieved remarkable fluency in short-form text generation, long-form narrative summarization remains a significant challenge. Modern models can summarize entire books or movies, but the resulting text often feels like a "list of events stapled together" rather than a cohesive story. This phenomenon, termed narrative incoherence, manifests as characters appearing without introduction, abrupt scene transitions, or internal factual contradictions.
Standard evaluation metrics such as ROUGE or BERTScore are largely "blind" to these discourse-level issues. Because they focus on word overlap or semantic similarity, they fail to penalize structural disarray. Furthermore, holistic human ratings (e.g., Likert scales) suffer from low inter-annotator agreement when applied to long summaries, as different readers prioritize different types of errors. To address these gaps, we introduce SNaC (Summary Narrative Coherence), a framework and dataset designed for fine-grained, span-level detection of coherence errors.
The SNaC Framework
The core of SNaC is a shift from holistic scoring to localized, categorized error detection. We developed a taxonomy of seven error types divided into two categories:
- Coherence Errors:
- CharE: Introduction of a new character without context.
- RefE: Reference to an event or object as if previously mentioned.
- SceneE: Abrupt or jarring transitions between narrative scenes.
- InconE: Factual inconsistencies or contradictions within the summary.
- Language & Fluency Errors: Including unnecessary repetition (RepE), ungrammatical or nonsensical phrasing (GramE), and unclear coreference (CorefE).
Segmented Annotation Workflow
To improve human agreement, SNaC utilizes a "step-by-step" annotation process. Instead of evaluating a full summary at once, annotators are presented with small segments (2–4 sentences) alongside the preceding context. This ensures that errors are identified relative to the established narrative flow.
This methodology significantly improved inter-annotator agreement, with Krippendorff’s $\alpha$ rising from a dismal 0.19 on holistic scales to 0.69 for specific categories like character errors.
Experimental Results
We applied the SNaC framework to summaries generated by various models, including GPT-3 (BOOK-175B, BOOK-6B) and BART-based models (MOVIE-BART). The resulting SNaC dataset contains 6.6k sentences and approximately 9.6k span-level annotations.
Quantifying Incoherence
Our analysis reveals that even state-of-the-art models like GPT-3 175B struggle with global planning. While they maintain high local fluency, they frequently fail at "connectedness." As shown in the distribution of errors, character introduction (CharE) and scene transitions (SceneE) are among the most prevalent issues across all model types.
Key findings from our experiments include:
- Metric Gameability: We demonstrated that a summary consisting merely of a list of named entities can achieve ROUGE scores comparable to human-written summaries, highlighting the inadequacy of overlap-based metrics.
- Synthetic Data Limitations: Models trained on "fake" errors (e.g., shuffled sentences) do not generalize well to the subtle narrative breaks produced by LLMs. In contrast, a T5 model fine-tuned on the SNaC dataset achieved high recall for real-world errors, such as Character (0.90) and Scene Transitions (0.82).
- Human vs. Machine: The SNaC-trained model often acted as a highly effective "red flag" system, identifying potential narrative breaks with higher recall than individual human annotators, though humans maintained higher precision.
Conclusion
SNaC provides a rigorous framework for diagnosing why AI summaries "lose the plot." By moving toward fine-grained, context-aware error detection, we can better evaluate the narrative capabilities of LLMs. Future work will focus on using SNaC to build "post-hoc" correction systems that can automatically identify and fix coherence gaps, ultimately leading to AI-generated summaries that are as narratively sound as they are fluent.
Summary updated