Modeling Global Coherence Beyond Local Sentence Transitions

16 concepts17 papers1 proposal

When we read a well-written document, it possesses an invisible "glue" known as text coherence. Coherence ensures that a sequence of sentences forms a unified, meaningful discourse rather than a random, disjointed list. Historically, computational models have evaluated coherence by looking through a magnifying glass: analyzing how smoothly one sentence transitions into the next.

However, a text can be locally flawless yet globally incoherent. Imagine an essay where every sentence connects smoothly to its immediate neighbor, but the text drifts from discussing canine anatomy to rocket propulsion, and finally to baking recipes. Locally, the transitions work; globally, the narrative is a "word salad." This discrepancy reveals the local-global coherence gap. Bridging this gap requires moving beyond immediate sentence pairs to capture the overarching logical organization, thematic progression, and macro-structural integrity of an entire document.

The Illusion of Local Coherence

Early breakthroughs in automated coherence modeling focused heavily on local syntax and entity transitions. In their seminal work, Barzilay et al. (2005) introduced the Entity Grid, a framework that tracks how specific entities (people, places, things) transition between grammatical roles (such as subject, object, or modifier) from one sentence to the next. By combining this grid with broader semantic similarity measures, Lapata et al. (2005) demonstrated a significantly higher correlation with human judgments of text quality than previous word-overlap metrics.

            Sentence 1: [Entity A: Subject] [Entity B: Object]
                                   │
                                   ▼
            Sentence 2: [Entity A: Object]  [Entity C: Subject]
                                   │
                                   ▼
            Sentence 3: [Entity D: Subject] [Entity C: Object]

The Entity Grid tracks local transitions across adjacent sentences, but it cannot see the overarching narrative arc.

While the Entity Grid successfully framed coherence as a machine-learning ranking problem, it operated on a strict sentence-to-sentence horizon. Around the same time, insights from Centering Theory highlighted that abrupt topic shifts—termed "Rough-Shifts"—were strong indicators of poor writing quality. As Miltsakaki et al. (2000) demonstrated, integrating these local shift metrics into automated essay scoring systems drastically improved their accuracy. Yet, the authors acknowledged an unresolved challenge: local entity flow does not inherently guarantee that a document possesses a broader thematic development or a structurally sound global organization.

Why Sliding Windows Fail

The transition to deep learning did not immediately solve this problem. Early neural coherence models, such as the one proposed by Li et al. (2014), replaced manual feature engineering with dense, distributed sentence representations. These models were trained to distinguish natural text from jumbled versions using a sliding window approach. While highly effective at capturing local semantic flow, sliding windows inherently suffer from a limited field of view. They struggle to model long-range dependencies, leaving them blind to the macro-structure of long-form content.

This blind spot creates severe vulnerabilities in downstream applications:

  • Adversarial Vulnerability: In automated essay scoring, models that prioritize local transitions can be easily fooled. Farag et al. (2018) showed that neural models optimized for local coherence often award high scores to adversarially crafted inputs—grammatical "word salads" that maintain smooth sentence-to-sentence transitions but completely lack a global logical argument or fail to address the prompt.
  • "Frankenstein" Summaries: In text summarization, simply extracting the most important sentences from a document often yields choppy, disconnected results. As Hu et al. (2018) observed, models need explicit mechanisms to capture long-range narrative flow and global consistency; otherwise, the summary fails to function as a cohesive whole.

Mapping the Macro-Structure

To capture the global rhetorical flow, researchers began exploring frameworks that treat text as a hierarchical or template-driven structure rather than a flat sequence of sentences.

One approach is discourse chunking. Instead of building complex, deep hierarchical trees, Sporleder et al. (2005) treated discourse as a sequence labeling task, identifying the core "nucleus" and peripheral "satellite" chunks of text across sentence boundaries. This allowed models to recognize the structural weight of different text segments.

For document-level evaluation, researchers turned to Rhetorical Structure Theory (RST). Traditional evaluation metrics in fields like machine translation (MT) typically operate at the sentence level, matching words or local clauses. To fix this, Guzmán et al. (2014) and Joty et al. (2017) introduced discourse-aware similarity measures based on RST. By modeling how clauses and sentences connect structurally across paragraphs, they demonstrated that capturing document-level rhetorical connections yields evaluation scores that align much more closely with human judgments.

                      [Global Document Node]
                           /          \
              [Nucleus: Main Idea]  [Satellite: Example]
                  /         \
       [Sentence 1]         [Sentence 2]

Hierarchical discourse structures map how localized sentences fulfill macro-level rhetorical roles.

Alternatively, some domains exhibit predictable, non-recurring thematic progressions—such as a Wikipedia article about a city always moving from History to Geography and then Demographics. To capture this, Chen et al. (2009) introduced a Bayesian topic model that treats document structure as a latent permutation governed by the generalized Mallows model. By learning these global templates, the model successfully captured long-range dependencies that standard Markovian local-transition models missed entirely.

Modern Neural Bridges

Recent research focuses on explicitly engineering global structural awareness into deep learning architectures, moving away from heavy manual annotations toward self-supervised and graph-based representations.

Self-Supervised Hierarchical Indirection

To scale these ideas without relying on expensive, human-labeled discourse trees, Jeon et al. (2020) combined the linguistic principles of Centering Theory with the representation power of Transformers. Their model automatically induces hierarchical discourse segments, allowing the network to evaluate whether a text functions as a unified global whole or merely a sequence of loosely related parts.

Discourse Corruption Pre-training

If local language models struggle with global organization because they are optimized for word- or sentence-level context, why not change the pre-training objective? Mim et al. (2020) introduced Discourse Corruption (DC). By intentionally shuffling paragraphs and sentences within essays and training a model to identify the specific type of structural disruption, the system naturally learns to recognize the high-level organizational properties of a well-structured argument.

   [Original Essay] ──► [Corrupt: Shuffle Paragraphs] ──► [Task: Identify Corruption Type]
                                                                   │
                                                                   ▼
                                                       Learns Global Macro-Structure

Graph-Based Structural Similarity

Other approaches transform documents into topological spaces to evaluate semantic and structural dependencies. For example, evaluating Discourse Representation Structures (DRSs) was historically bottlenecked by computational complexity. Liu et al. (2020) bypassed this with DSCORER, a metric that converts DRSs into graphs to efficiently capture long-range semantic dependencies that local clause-matching metrics miss.

Taking this further, Liu et al. (2023) modeled structural similarities between different documents by constructing large-scale heterogeneous graphs processed via Graph Convolutional Networks (GCNs). They noted, however, a lingering constraint: decomposing document structures into fixed-size $k$-node subgraphs can still fail to capture the largest, most complex organizational patterns that emerge across massive spans of text.

Ultimately, the evolution of coherence modeling shows that text quality cannot be measured purely by the smoothness of a local sliding window. True text understanding requires architectures that can simultaneously track the immediate semantic flow between adjacent sentences and the overarching, macro-level blueprint of the entire discourse.

Key Foundational Work

The Role of Centering Theory's Rough-Shift in the Teaching and Evaluation of Writing Skills (2000) — contributed 1 concept elements to this direction.
Automatic Evaluation of Text Coherence: Models and Representations (2005) — contributed 1 concept elements to this direction.
Modeling Local Coherence: An Entity-Based Approach (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_local_global_coherence_gap_ca506c84,
  title = {local-global coherence gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-global-coherence-gap/rdr_ca506c8479814055a796dca8054f79b2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_local_global_coherence_gap_ca506c84_v1,
  title = {local-global coherence gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-global-coherence-gap/rdr_ca506c8479814055a796dca8054f79b2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake