When we read a well-written document, we perceive a "global" structure—a logical progression from introduction to conclusion. However, a growing body of research suggests that this high-level coherence is not necessarily a top-down architectural feat. Instead, it is often an emergent property of successful local transitions between adjacent sentences. By focusing on how one sentence leads to the next, researchers have developed efficient ways to order information, evaluate text quality, and even detect factual hallucinations.
The Markovian Assumption of Discourse
The foundational intuition behind local coherence is that a text can be modeled as a sequence of transitions. In Probabilistic Text Structuring: Experiments with Sentence Ordering, Lapata treats sentence ordering as a Markov process, where the probability of any given sentence depends primarily on the one immediately preceding it. This simplification transforms the daunting task of global planning into a series of local optimization problems.
This "bottom-up" philosophy is echoed in the work of Okazaki et al., who argue that satisfying local constraints between adjacent pairs is sufficient to construct a globally coherent document. Their research suggests that complex global planning can be replaced by iterative local optimization, where the system simply ensures that every sentence "fits" with its immediate neighbors.
Cohesion Blocks and Narrative Flow
In practice, humans do not just link individual sentences; they group related ideas into "cohesion blocks." Barzilay et al. observed that while there is rarely a single "correct" global order for a summary, maintaining these local thematic clusters is critical for readability.
This focus on local flow has proven more robust than global modeling in several high-stakes tasks:
- Sentence Insertion: When placing a stray sentence back into a paragraph, Wu et al. found that local linguistic cohesion is a more reliable indicator of the correct position than broad global context.
- Domain Transfer: Xu et al. demonstrated that models focusing exclusively on local transitions can transfer their "understanding" of coherence across vastly different domains—such as from Wikipedia to financial news—more effectively than architectures designed for document-wide dependencies.
- Clinical Diagnostics: Gkoumas et al. used the predictability (perplexity) of local sentence transitions to monitor cognitive decline, showing that a breakdown in thematic continuity between sentences is a strong digital marker for Alzheimer’s.
Adjacency as a Proxy for Truth
One of the most compelling applications of local coherence modeling is in identifying "hallucinations" in abstractive summarization. Gabriel et al. introduced Co-opNet, which uses a discriminator to predict the likelihood that two sentences should appear consecutively.
The authors argue that sentence adjacency prediction serves as a powerful proxy for factual consistency. The intuition is simple: when a model begins to hallucinate, the logical thread of the narrative usually snaps. By forcing a model to maintain a coherent narrative flow, Gabriel et al. showed that the likelihood of generating unsupported or disjointed information decreases. Similarly, Strube et al. found a strong correlation between a text’s discourse coherence—specifically how entities transition between sentences—and its factual reliability.
Rethinking How We Measure Quality
If local flow is the primary driver of perceived quality, our evaluation metrics should reflect that. Traditional metrics like Kendall’s $\tau$ often penalize models for "global" errors, such as shifting a perfectly coherent block of text to the wrong part of a document.
Recent work proposes a shift toward metrics that reward local continuity:
- Weighted Longest Common Subsequence (WLCS-l): Liu & Zeng argue that this metric aligns better with human judgment because it rewards the preservation of consecutive, overlapping sequences rather than absolute positioning.
- Blocked Kendall’s $\tau$: Guz & Carenini proposed merging correctly ordered sub-sequences into "blocks" before scoring, ensuring that models aren't over-penalized for global shifts if the internal local structure remains intact.
- GRUEN: Zhu developed this unsupervised metric to penalize large semantic gaps or illogical ordering between consecutive sentences, reflecting the principle that topical focus is a local phenomenon.
By treating coherence as a chain of successful local transitions, these researchers have provided a more computationally tractable and human-aligned way to generate and evaluate natural language.
Key Foundational Work
Probabilistic Text Structuring: Experiments with Sentence Ordering (2003) — contributed 1 concept elements to this direction.
Sentence Ordering in Multidocument Summarization (2001) — contributed 1 concept elements to this direction.