Abstract
This paper explores the sentence ordering problem as a means to model text coherence. We propose a "set-to-sequence" deep learning framework that learns to arrange a shuffled set of sentences into a logically consistent paragraph. By training on this self-supervised task, the model develops deep semantic representations of text that capture discourse structure. Our results on standard benchmarks and new scientific abstract datasets show that the model achieves state-of-the-art performance in sentence ordering and produces embeddings that rival established pre-training methods for downstream NLP tasks.
Introduction
In Natural Language Processing (NLP), "coherence" refers to the logical connections that make a text meaningful. While modern models excel at local word prediction, understanding high-level logical flow remains a significant challenge. Traditional approaches to coherence modeling often relied on handcrafted features, such as tracking entity transitions from sentence to sentence.
This work addresses the Sentence Ordering Problem: given an unordered "bag" of sentences, the goal is to determine the single most coherent sequence. Solving this is critical for applications like multi-document summarization, where information from disparate sources must be synthesized into a readable and logically structured report.
Methods: The Set-to-Sequence Framework
The proposed model utilizes a Set-to-Sequence (set2seq) architecture designed to handle unordered inputs and produce ordered outputs. The framework consists of three primary components:
- Sentence Encoder: A word-level Recurrent Neural Network (RNN) that transforms each individual sentence into a fixed-length vector embedding.
- Set Encoder: This component acts as the "brain" of the model. Using an attention mechanism, it processes the entire pool of sentence embeddings simultaneously. Through multiple "read cycles," it constructs a global context of the paragraph's overall meaning, independent of the initial input order.
- Pointer Network Decoder: Instead of generating new text, the decoder "points" to specific sentences in the original pool to construct the final sequence. It leverages the global context from the encoder to decide which sentence logically follows the current sequence.
To enhance robustness, the authors introduced Contrastive Sentences—random "distractor" sentences added during training—to force the model to distinguish relevant content from noise.
Experimental Results
The model was evaluated on two distinct tasks: order discrimination (identifying original vs. shuffled text) and sentence ordering (reconstructing the original sequence).
Order Discrimination and Sentence Ordering
On standard benchmarks like the "Accidents" and "Earthquakes" datasets, the model achieved near-perfect accuracy (94.4% and 99.7% respectively), significantly outperforming previous baselines like the Entity-Grid and Window Network.
For the more complex sentence ordering task, the authors introduced benchmarks using scientific abstracts from NIPS, ACL, and NSF. Scientific abstracts are ideal for this task because they typically follow a rigid logical structure (Introduction $\rightarrow$ Problem $\rightarrow$ Method $\rightarrow$ Results). On NIPS abstracts, the model achieved an accuracy of 51.55% and a Kendall’s tau of 0.72, far exceeding random chance and previous sequence-to-sequence baselines.
Visualizing Discourse Logic
A key finding of this research is that the model naturally learns to identify "clue words" (e.g., First, Second, We argue) that signal discourse transitions. t-SNE visualizations of the learned sentence embeddings reveal that the model organizes sentences based on their relative position within a document, effectively "discovering" the concept of transition logic without explicit supervision.
Conclusion
The set-to-sequence framework demonstrates that sentence ordering is a powerful self-supervised task for learning discourse structure. By focusing on global context rather than local windows, the model achieves state-of-the-art results in coherence modeling. Future work will explore scaling this approach to longer documents and incorporating explicit linguistic features to better handle out-of-vocabulary terms.
Summary updated