TL;DR
SNaC (Summary Narrative Coherence) is a new framework and dataset designed to detect and localize fine-grained coherence errors in long-form narrative summaries. By moving beyond holistic "quality" scores to specific span-level annotations, the authors demonstrate that current state-of-the-art models frequently fail at narrative flow—and that their new classifier can identify these failures far better than standard metrics like ROUGE.
What Problem Does This Paper Address?
As Large Language Models (LLMs) scale, they have become remarkably good at generating fluent, on-topic sentences. However, when tasked with summarizing long documents like books or movie scripts, the "big picture" often falls apart. A summary might be grammatically perfect but narratively nonsensical—introducing characters out of nowhere, jumping abruptly between scenes, or referencing events that were never mentioned.
Current evaluation methods are ill-equipped for this. Automatic metrics like ROUGE focus on word overlap, which doesn't capture flow. Human evaluation typically relies on Likert scales (e.g., "rate coherence from 1-5"), which are highly subjective and result in low inter-annotator agreement for long texts. The authors argue that we need a way to pinpoint where and why a narrative breaks down.
Methods
The authors introduce a structured approach to coherence evaluation:
- Error Taxonomy: They developed a schema of seven error types, split into two categories:
- Coherence Errors: New character without introduction (CharE), missing reference to an event/object (RefE), abrupt scene transitions (SceneE), and logical inconsistencies (InconE).
- Language & Fluency Errors: Unnecessary repetition (RepE), ungrammatical/nonsensical text (GramE), and unclear coreference (CorefE).
- Annotation Protocol: Instead of asking workers to rate a 500-word summary at once, they decomposed the task. Annotators evaluated small segments (2-4 sentences) while having access to the preceding context. They highlighted specific spans and assigned error labels.
- Automated Detection: The authors fine-tuned a T5-Large model on their collected annotations to automatically detect and localize these errors. They compared this "SNaC-trained" model against unsupervised methods (like GPT-2 perplexity) and models trained on synthetic data (artificially corrupted text).
Datasets
The study utilized several key sources to build the SNaC Dataset:
- Book Summaries: Generated by OpenAI’s GPT-3 (175B and 6B versions) using the BookSum dataset.
- Movie Summaries: Generated using a BART-based model (Summ^N) on the TRIPOD dataset.
- The SNaC Dataset itself: Contains 9.6k span-level annotations across 160 summaries (6.6k sentences). This includes both expert annotations and crowdsourced data from trained Mechanical Turk workers.
Key Insights and Ideas
- Holistic Scores are "Gameable": The authors found that even if you randomly shuffle the sentences of a summary or repeat 50% of them, ROUGE scores barely budge. This highlights a dangerous blind spot in how we currently measure summarization progress.
- Granularity Improves Agreement: While humans struggle to agree on a single 1-5 coherence score for a long summary, they show much higher agreement when asked to identify specific spans where a new character appears without introduction.
- Narrative Summarization is Unique: The errors found in summaries (like missing context) are fundamentally different from errors in open-ended story generation (which are dominated by repetition). This justifies the need for a summarization-specific taxonomy.
- Synthetic Data is a Poor Substitute: Models trained on "fake" errors (like random sentence swapping) performed poorly on real-world model failures. Training on actual human-identified errors is essential for building reliable coherence classifiers.
Key Results
- Model Performance: The fine-tuned T5 model achieved strong results in localizing errors, particularly for CharE (F1 of 0.80) and SceneE (F1 of 0.40-0.53).
- Human vs. Machine: Interestingly, the trained model often achieved higher recall than individual human annotators. Because the model was trained on the aggregated knowledge of multiple annotators, it became more sensitive to subtle errors that a single human might overlook.
- Error Prevalence: The analysis revealed that state-of-the-art models are still struggling significantly; each generated summary contained an average of ~30 coherence errors of varying severity.
Limitations and Future Work
- Domain & Language: The study is limited to English-language narratives (books and movies). The taxonomy might need adjustments for other domains like government reports or technical documentation.
- Subjectivity of "Missing Info": The
RefE(Missing Reference) category remains the hardest for both humans and models to agree on, as "what context is necessary" can be subjective. - Future Applications: The authors suggest that SNaC could be used not just for evaluation, but for post-hoc correction. A model could detect a coherence error (e.g., a missing character intro) and then prompt a secondary model to "fix" that specific span by adding the necessary context.
For those interested in exploring the data or models, the authors have released their work at github.com/tagoyal/snac.
Summary updated