When we read a well-written essay or a news report, we perceive a "flow"—a logical progression that makes the text feel like a single, unified entity rather than a pile of disconnected facts. Intuitively, we might assume this coherence stems from a grand architectural plan or a global document structure. However, a growing body of research suggests that document-level coherence is primarily an emergent property of local transitions.
If every sentence connects logically to its immediate neighbor, the global structure often takes care of itself. This "bottom-up" perspective transforms the complex problem of document planning into a series of manageable, local optimizations.
The Markovian Assumption of Discourse
The foundational intuition for this direction is that text can be modeled as a Markov process. In this view, the probability of a sentence appearing is conditioned primarily on the sentence immediately preceding it. Mirella Lapata demonstrated that this simplification—treating sentence ordering as a sequence of local transitions—is remarkably effective for probabilistic text structuring.
This principle suggests that satisfying local constraints between adjacent pairs is sufficient to construct a globally coherent document. Naoaki Okazaki and colleagues applied this "bottom-up" approach to multi-document summarization, using an SVM classifier to combine criteria like chronology and topical similarity. They found that by focusing on whether sentence B follows sentence A, they could build summaries that humans found significantly more readable than those produced by global planning methods.
Tracking Entities and Topics
How do we define a "good" transition? Two primary mechanisms emerge from the literature: Entity Centering and Topic Continuity.
- Centering Theory: This linguistic framework tracks how "entities" (nouns and pronouns) transition between sentences. Masaaki Nishino leveraged this to improve multi-document summarization, showing that tracking entity transitions is a lightweight but powerful proxy for coherence. Similarly, Michael Strube incorporated these principles into DiscoScore, a metric that outperforms standard benchmarks by evaluating how consistently a text refers to its central subjects.
- Topic Chains: Beyond specific entities, coherence relies on a "continuity of senses." Deyi Xiong introduced a topic-based coherence model for machine translation that predicts a sequence of sentence-level topics. Their work showed that a short-range history—looking back only two sentences—provides the most relevant context for maintaining a logical thread.
Local Adjacency as the Strongest Signal
A recurring finding across different tasks—from argument mining to sentence insertion—is that local signals are often more predictive than global ones.
In the "Sentence Insertion" problem (common in TOEFL exams), Stan Z. Li and Fang Wu found that Graph Neural Networks performed best when they prioritized the immediate neighbors of the insertion slot. Their model, InsertGNN, isolates subgraphs of just three sentences to capture fine-grained linguistic transitions that global document features often miss.
This dominance of local context is also evident in stylistic analysis. René Knaebel and Manfred Stede discovered that the "fingerprint" of a writing style (e.g., news vs. scientific abstracts) is most visible in the immediate transitions between rhetorical moves, such as moving from a Contrast to a Cause.
Coherence as a Guardrail for Truth
Perhaps the most surprising benefit of focusing on local coherence is its impact on factual consistency. In abstractive summarization, models often "hallucinate" or invent facts. Saadia Gabriel and Aslı Çelikyılmaz found that enforcing a rigorous discourse structure acts as a proxy for truthfulness.
When a model is constrained to follow a logical narrative arc—ensuring that sentence B follows logically from sentence A—it has less "room" to insert fabricated information. Alexander Spangher demonstrated this with DiscoSum, where forcing Large Language Models to follow specific functional sequences (like "Hook" → "Context") significantly reduced hallucinations.
Generalizing Through Locality
Finally, focusing on local transitions makes models more robust. Peng Xu and Jackie CK Cheung showed that while global document structures vary wildly between domains (e.g., a Wikipedia article vs. a Wall Street Journal report), the fundamental "rules" of how two sentences connect remain relatively stable. By training a model to score the coherence of adjacent sentence pairs, they created a system that could generalize its understanding of "good writing" across entirely different genres of text.
In summary, the research suggests that we should view a document not as a single monolithic structure, but as a chain of local dependencies. By perfecting the links between adjacent ideas, we naturally arrive at documents that are coherent, readable, and factually grounded.