A Model of Coherence Based on Distributed Sentence Representation

Jiwei Li, Eduard H. HovyEMNLP 2014 · OpenReview

Abstract

Coherence is what makes a multi-sentence text meaningful, both logically and syntactically. To solve the challenge of ordering a set of sentences into coherent order, existing approaches focus mostly on defining and using sophisticated features to capture the cross-sentence argumentation logic and syntactic relationships. However, both argumentation semantics and cross-sentence syntax (such as coreference and tense rules) are very hard to formalize. In this paper, we introduce a neural network model for the coherence task based on distributed sentence representation. The proposed approach learns a syntactico-semantic representation for sentences automatically, using either recurrent or recursive neural networks. The architecture obviates the need for feature engineering and learns sentence representations which are, to some extent, able to capture the 'rules' governing coherent sentence structure. The proposed approach outperforms existing baselines and generates state-of-the-art performance in standard coherence evaluation tasks.

Introduction

In natural language processing, coherence is the essential quality that distinguishes a structured paragraph from a random collection of sentences. Traditionally, modeling coherence relied on formal logic frameworks like Rhetorical Structure Theory or tracking specific "entities" (nouns) across sentences, as seen in the Entity Grid model.

However, these traditional methods suffer from a heavy reliance on hand-crafted features. Researchers had to explicitly define what the model should look for—such as coreference links or specific syntactic patterns. These features are difficult to formalize and often depend on error-prone upstream tools like parsers. This paper proposes a shift toward learning these coherence rules automatically from raw data using distributed representations.

Example of coherent text versus a non-coherent version created via random sentence replacement.

Methodology

The authors introduce a two-step neural architecture designed to capture both the internal structure of sentences and their relationship to one another.

Sentence Representation

Each sentence is transformed into a fixed-length vector ($h_s$). The study explored two primary methods for this transformation:

  1. Recurrent Neural Networks (RNN): These process words sequentially, where each word's representation is influenced by its predecessors.
  2. Recursive Neural Networks: These utilize a sentence's parse tree to combine word vectors hierarchically from the bottom up, capturing grammatical structure more explicitly.

Coherence Modeling

To evaluate text coherence, the model examines a "window" of $L$ sentences (typically 3). These sentence vectors are concatenated and passed through a three-layer neural network. The model is trained discriminatively: it learns to distinguish a window of sentences from a real article (positive example) from a window where a sentence has been replaced by a random one (negative example). This approach allows the network to implicitly learn the "rules" of logical flow and tense consistency without manual feature engineering.

Experimental Setup

The model was evaluated on two distinct tasks:

  • Sentence Ordering: Using reports on Airplane Accidents (NTSB) and Earthquakes (Associated Press). The goal is to correctly identify the original, coherent sequence from a set of permutations.
  • Readability Assessment: Comparing Encyclopedia Britannica with Britannica Elementary. The hypothesis is that the "Elementary" version, written for children, is inherently more coherent and accessible.

Key Results

The neural models demonstrated significant improvements over traditional baselines. In the sentence ordering task, the Recursive model achieved an accuracy of 0.976 on the Earthquake dataset, substantially outperforming the previous state-of-the-art Entity Grid model (0.872).

Accuracy of various models on the Accident and Earthquake sentence ordering datasets.

On the more linguistically diverse Accident dataset, the Recursive model reached 0.864 accuracy. In readability assessment, it achieved 0.828, surpassing both the "Graph Approach" and standard "S&O" readability features.

Discussion and Conclusion

A key insight from this research is that the Recursive model consistently outperformed the Recurrent model, suggesting that internal grammatical structure is vital for understanding discourse-level coherence. Furthermore, the model's exceptional performance on the Earthquake dataset highlights its ability to "memorize" domain-specific structures (e.g., location $\rightarrow$ intensity $\rightarrow$ damage) through distributed representations.

While the "random replacement" training strategy is a coarse proxy for coherence, it effectively eliminates the need for manual feature engineering. Future work may focus on more subtle "negative" examples—such as sentences that are on-topic but slightly out of order—and extending the model from local "windows" to global document-level coherence.

Summary updated

Abstract

Coherence is what makes a multi-sentence text meaningful, both logically and syntactically. To solve the challenge of ordering a set of sentences into coherent order, existing approaches focus mostly on defining and using sophisticated features to capture the cross-sentence argumentation logic and syntactic relationships. But both argumentation semantics and crosssentence syntax (such as coreference and tense rules) are very hard to formalize. In this paper, we introduce a neural network model for the coherence task based on distributed sentence representation. The proposed approach learns a syntacticosemantic representation for sentences automatically, using either recurrent or recursive neural networks. The architecture obviated the need for feature engineering, and learns sentence representations, which are to some extent able to capture the ‘rules’ governing coherent sentence structure. The proposed approach outperforms existing baselines and generates the stateof-art performance in standard coherence evaluation tasks 1 .
BibTeX
@inproceedings{li2014model,
  author = {Jiwei Li and Eduard H. Hovy},
  title = {{A Model of Coherence Based on Distributed Sentence Representation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  url = {https://openreview.net/forum?id=Hy-jiffu-B},
}
Figure