Narrative Incoherence Detection

Deng Cai, Yizhe Zhang, Yichen Huang, Wai Lam, Bill DolanarXiv 2020 · arXiv

Overview

Modern large language models excel at generating fluent sentences but often struggle to maintain narrative coherence over long passages. This paper introduces "Narrative Incoherence Detection," a task focused on identifying where a story's logical flow breaks down. The authors evaluate two primary architectures—token-level and sentence-level Transformers—finding that sentence-level models offer superior efficiency and performance on longer texts when combined with large-scale pre-training.

Defining Narrative Incoherence

The core challenge addressed is the tendency of generative models to "lose the plot," resulting in repetitions, contradictions, or logical leaps. To benchmark this, the authors define two specific corruption types:

  1. Missing Sentence Detection (MSD): Identifying a location where a necessary bridging sentence has been omitted.
  2. Discordant Sentence Detection (DSD): Identifying a sentence that, while grammatically correct, is semantically out of place.
Examples of coherent vs. corrupted narratives illustrating Missing Sentence Detection (MSD) and Discordant Sentence Detection (DSD).

Modeling Approaches

The study compares two distinct paradigms for processing multi-sentence narratives:

Token-Level Approach

This method follows standard NLP practices by concatenating all sentences into a single sequence separated by [SEP] tokens. The entire sequence is processed by BERT, and a classifier operates on the contextualized representations of the [SEP] tokens or individual sentences to detect incoherence.

The token-level architecture uses BERT to process a flat sequence of tokens and classify incoherence at specific positions.

Sentence-Level Approach

In contrast, the sentence-level approach treats sentences as atomic units. It uses fixed sentence embeddings (extracted via BERT) as inputs to a dedicated sentence-level Transformer. This approach is significantly more computationally efficient, scaling linearly with token length per sentence rather than quadratically across the entire document.

To enhance the sentence-level model, the authors introduced:

  • Large-Scale Pre-training: Training the sentence Transformer on a massive corpus of stories.
  • Semantic Matching (SM): A multi-task objective where the model predicts the latent embedding of the missing or discordant sentence.
  • DAE Fine-Tuning: Using a Denoising Autoencoder to map latent representations back to text via a GPT-2 decoder.

Datasets and Evaluation

The models were trained and tested on two datasets:

  • TIMETRAVEL: 5-sentence commonsense stories derived from the ROCStories dataset.
  • TRIPADVISOR: 8-sentence segments from hotel reviews.

Training data was automatically generated by randomly removing sentences (MSD) or replacing them with semantically similar but incongruous ones (DSD). For evaluation, human crowd-workers verified the test sets to ensure the "corruptions" were truly incoherent.

Key Findings

  • Granularity vs. Efficiency: Token-level models are slightly more accurate on short texts (e.g., 81.2% vs 80.5% on TimeTravel MSD) because they capture fine-grained word dependencies. However, their computational cost scales quadratically ($O(N^2L^2)$).
  • Scalability: Sentence-level models scale much better ($O(N^2 + NL^2)$) and outperform token-level models on longer narratives like TripAdvisor (80.4% vs 74.8% for DSD).
  • Speed: On the TripAdvisor dataset, the sentence-level model achieved a >21x speedup, processing ~860 paragraphs per second compared to ~40 for the token-level model.
  • Pre-training Benefits: Pre-training on external story corpora allows sentence-level models to bridge the performance gap with token-level models while maintaining their efficiency advantage.

Conclusion

Narrative Incoherence Detection serves as a vital benchmark for inter-sentential understanding. While token-level models are precise for short contexts, sentence-level Transformers are the practical choice for long-form text analysis. Future applications include building AI proofreading tools that repair logical flow and reducing hallucinations in long-form LLM generation.

Summary updated

Abstract

We propose the task of narrative incoherence detection as a new arena for inter-sentential semantic understanding: Given a multi-sentence narrative, decide whether there exist any semantic discrepancies in the narrative flow. Specifically, we focus on the missing sentence and discordant sentence detection. Despite its simple setup, this task is challenging as the model needs to understand and analyze a multi-sentence narrative, and predict incoherence at the sentence level. As an initial step towards this task, we implement several baselines either directly analyzing the raw text (\textit{token-level}) or analyzing learned sentence representations (\textit{sentence-level}). We observe that while token-level modeling has better performance when the input contains fewer sentences, sentence-level modeling performs better on longer narratives and possesses an advantage in efficiency and flexibility. Pre-training on large-scale data and auxiliary sentence prediction training objective further boost the detection performance of the sentence-level model.
BibTeX
@misc{cai2020narrative,
  author = {Deng Cai and Yizhe Zhang and Yichen Huang and Wai Lam and Bill Dolan},
  title = {{Narrative Incoherence Detection}},
  year = {2020},
  eprint = {2012.11157},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.11157},
}
Figure