Evaluating Document Coherence Modelling

Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Timothy Baldwin, Jianzhong Qi

Introduction

Large Language Models (LLMs) have demonstrated remarkable proficiency in generating grammatically correct and locally coherent sentences. However, their ability to maintain discourse coherence—the logical and rhetorical "glue" that binds a sequence of sentences into a unified document—remains a subject of intense research. While LLMs can mimic the style of a news report or a Wikipedia entry, it is unclear whether they truly grasp the underlying logic of discourse or are simply leveraging local statistical patterns.

The Sentence Intrusion Task

To evaluate coherence, the authors propose a "Sentence Intrusion Detection" task. Unlike artificial tasks like sentence shuffling, this approach presents a more realistic challenge: identifying a single "intruder" sentence within an otherwise coherent document. The intruder is not random; it is topically similar to the original context but logically out of place. This requires models to understand complex rhetorical relations such as contrast, narration, and elaboration.

The task is divided into two levels of granularity:

  1. Coarse-grained: A binary classification task to determine if a document contains an intruder.
  2. Fine-grained: A span identification task to locate the specific intruder sentence.

To ensure the benchmark is sufficiently challenging, the researchers employed adversarial filtering. They used a fine-tuned XLNet model to discard "easy" examples, retaining only those where the intruder is topically relevant but logically inconsistent.

The INSteD Dataset

The study introduces the Intruder Sentence Detection (INSteD) dataset, a large-scale benchmark comprising over 170,000 documents. The dataset is sourced from two distinct domains: Wikipedia and CNN News.

Table 1: Statistics for the INSteD dataset, showing the distribution of documents and average lengths across Wikipedia and CNN sources.

As shown in the table above, the dataset provides a robust foundation for testing model generalization across different writing styles and document structures, with an average document length of approximately five sentences.

Experimental Results and the "Cross-Domain Cliff"

The researchers evaluated a variety of state-of-the-art models, including BERT, XLNet, RoBERTa, and ALBERT, against human performance.

Table 2: Performance comparison of various models on the INSteD task. Bold values indicate top performance, highlighting the significant gap between in-domain and cross-domain accuracy.

Key Findings:

  • Top Performance: ALBERT-xxLarge emerged as the strongest model, achieving an accuracy of 81.7% on Wikipedia and a staggering 96.9% on CNN in-domain.
  • The Cross-Domain Cliff: A significant performance drop was observed when models were moved to new domains. For instance, when testing on Wikipedia documents, a model trained on CNN data saw its accuracy drop from the in-domain 81.7% to just 66.6%. This suggests that models may be memorizing domain-specific "templates" of flow rather than learning universal principles of discourse.
  • AI as a Fact-Checker: On Wikipedia tasks, models occasionally outperformed humans (who averaged 66.6% accuracy). This is likely because the models were pretrained on Wikipedia; they often identified intruders by spotting factual inaccuracies based on memory rather than logical breaks.

Analysis of Incoherence Types

Linguistic probes were used to dissect model performance across different types of "glitches," such as gender pronoun flips, tense shifts, and numerical inconsistencies. The results indicate that while models are adept at detecting breaks in "information flow" (e.g., a sudden change in topic), they struggle with subtle logical inconsistencies, such as demonstrative pronouns referring to non-existent antecedents.

Conclusion and Future Directions

While top-tier models show promise, their reliance on domain-specific patterns and factual memory highlights a gap in true discourse understanding. The massive performance drop across domains remains a major challenge. Future work must focus on training models that understand the fundamental principles of discourse, regardless of whether they are reading a recipe, a news report, or a legal brief.

Summary updated

BibTeX
@article{shen2021evaluating,
  author = {Aili Shen and Meladel Mistica and Bahar Salehi and Hang Li and Timothy Baldwin and Jianzhong Qi},
  title = {{Evaluating Document Coherence Modelling}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00388},
}
Figure