Abstract
Coherence is what makes a multi-sentence text meaningful, both logically and syntactically. To solve the challenge of ordering a set of sentences into coherent order, existing approaches focus mostly on defining and using sophisticated features to capture the cross-sentence argumentation logic and syntactic relationships. However, both argumentation semantics and cross-sentence syntax (such as coreference and tense rules) are very hard to formalize. In this paper, we introduce a neural network model for the coherence task based on distributed sentence representation. The proposed approach learns a syntactico-semantic representation for sentences automatically, using either recurrent or recursive neural networks. The architecture obviates the need for feature engineering and learns sentence representations which are, to some extent, able to capture the 'rules' governing coherent sentence structure. The proposed approach outperforms existing baselines and generates state-of-the-art performance in standard coherence evaluation tasks.
Introduction
In natural language processing, coherence is the essential quality that distinguishes a structured paragraph from a random collection of sentences. Traditionally, modeling coherence relied on formal logic frameworks like Rhetorical Structure Theory or tracking specific "entities" (nouns) across sentences, as seen in the Entity Grid model.
However, these traditional methods suffer from a heavy reliance on hand-crafted features. Researchers had to explicitly define what the model should look for—such as coreference links or specific syntactic patterns. These features are difficult to formalize and often depend on error-prone upstream tools like parsers. This paper proposes a shift toward learning these coherence rules automatically from raw data using distributed representations.
Methodology
The authors introduce a two-step neural architecture designed to capture both the internal structure of sentences and their relationship to one another.
Sentence Representation
Each sentence is transformed into a fixed-length vector ($h_s$). The study explored two primary methods for this transformation:
- Recurrent Neural Networks (RNN): These process words sequentially, where each word's representation is influenced by its predecessors.
- Recursive Neural Networks: These utilize a sentence's parse tree to combine word vectors hierarchically from the bottom up, capturing grammatical structure more explicitly.
Coherence Modeling
To evaluate text coherence, the model examines a "window" of $L$ sentences (typically 3). These sentence vectors are concatenated and passed through a three-layer neural network. The model is trained discriminatively: it learns to distinguish a window of sentences from a real article (positive example) from a window where a sentence has been replaced by a random one (negative example). This approach allows the network to implicitly learn the "rules" of logical flow and tense consistency without manual feature engineering.
Experimental Setup
The model was evaluated on two distinct tasks:
- Sentence Ordering: Using reports on Airplane Accidents (NTSB) and Earthquakes (Associated Press). The goal is to correctly identify the original, coherent sequence from a set of permutations.
- Readability Assessment: Comparing Encyclopedia Britannica with Britannica Elementary. The hypothesis is that the "Elementary" version, written for children, is inherently more coherent and accessible.
Key Results
The neural models demonstrated significant improvements over traditional baselines. In the sentence ordering task, the Recursive model achieved an accuracy of 0.976 on the Earthquake dataset, substantially outperforming the previous state-of-the-art Entity Grid model (0.872).
On the more linguistically diverse Accident dataset, the Recursive model reached 0.864 accuracy. In readability assessment, it achieved 0.828, surpassing both the "Graph Approach" and standard "S&O" readability features.
Discussion and Conclusion
A key insight from this research is that the Recursive model consistently outperformed the Recurrent model, suggesting that internal grammatical structure is vital for understanding discourse-level coherence. Furthermore, the model's exceptional performance on the Earthquake dataset highlights its ability to "memorize" domain-specific structures (e.g., location $\rightarrow$ intensity $\rightarrow$ damage) through distributed representations.
While the "random replacement" training strategy is a coarse proxy for coherence, it effectively eliminates the need for manual feature engineering. Future work may focus on more subtle "negative" examples—such as sentences that are on-topic but slightly out of order—and extending the model from local "windows" to global document-level coherence.
Summary updated