Overview
Modern large language models excel at generating fluent sentences but often struggle to maintain narrative coherence over long passages. This paper introduces "Narrative Incoherence Detection," a task focused on identifying where a story's logical flow breaks down. The authors evaluate two primary architectures—token-level and sentence-level Transformers—finding that sentence-level models offer superior efficiency and performance on longer texts when combined with large-scale pre-training.
Defining Narrative Incoherence
The core challenge addressed is the tendency of generative models to "lose the plot," resulting in repetitions, contradictions, or logical leaps. To benchmark this, the authors define two specific corruption types:
- Missing Sentence Detection (MSD): Identifying a location where a necessary bridging sentence has been omitted.
- Discordant Sentence Detection (DSD): Identifying a sentence that, while grammatically correct, is semantically out of place.
Modeling Approaches
The study compares two distinct paradigms for processing multi-sentence narratives:
Token-Level Approach
This method follows standard NLP practices by concatenating all sentences into a single sequence separated by [SEP] tokens. The entire sequence is processed by BERT, and a classifier operates on the contextualized representations of the [SEP] tokens or individual sentences to detect incoherence.
Sentence-Level Approach
In contrast, the sentence-level approach treats sentences as atomic units. It uses fixed sentence embeddings (extracted via BERT) as inputs to a dedicated sentence-level Transformer. This approach is significantly more computationally efficient, scaling linearly with token length per sentence rather than quadratically across the entire document.
To enhance the sentence-level model, the authors introduced:
- Large-Scale Pre-training: Training the sentence Transformer on a massive corpus of stories.
- Semantic Matching (SM): A multi-task objective where the model predicts the latent embedding of the missing or discordant sentence.
- DAE Fine-Tuning: Using a Denoising Autoencoder to map latent representations back to text via a GPT-2 decoder.
Datasets and Evaluation
The models were trained and tested on two datasets:
- TIMETRAVEL: 5-sentence commonsense stories derived from the ROCStories dataset.
- TRIPADVISOR: 8-sentence segments from hotel reviews.
Training data was automatically generated by randomly removing sentences (MSD) or replacing them with semantically similar but incongruous ones (DSD). For evaluation, human crowd-workers verified the test sets to ensure the "corruptions" were truly incoherent.
Key Findings
- Granularity vs. Efficiency: Token-level models are slightly more accurate on short texts (e.g., 81.2% vs 80.5% on TimeTravel MSD) because they capture fine-grained word dependencies. However, their computational cost scales quadratically ($O(N^2L^2)$).
- Scalability: Sentence-level models scale much better ($O(N^2 + NL^2)$) and outperform token-level models on longer narratives like TripAdvisor (80.4% vs 74.8% for DSD).
- Speed: On the TripAdvisor dataset, the sentence-level model achieved a >21x speedup, processing ~860 paragraphs per second compared to ~40 for the token-level model.
- Pre-training Benefits: Pre-training on external story corpora allows sentence-level models to bridge the performance gap with token-level models while maintaining their efficiency advantage.
Conclusion
Narrative Incoherence Detection serves as a vital benchmark for inter-sentential understanding. While token-level models are precise for short contexts, sentence-level Transformers are the practical choice for long-form text analysis. Future applications include building AI proofreading tools that repair logical flow and reducing hallucinations in long-form LLM generation.
Summary updated