Deep Attentive Ranking Networks for Learning to Order Sentences

Pawan Kumar, Dhanajit Brahma, Harish Karnick, Piyush Rai

Abstract

We present an attention-based ranking framework for learning to order sentences given a paragraph. Our framework is built on a bidirectional sentence encoder and a self-attention based transformer network to obtain an input order invariant representation of paragraphs. Moreover, it allows seamless training using a variety of ranking based loss functions, such as pointwise, pairwise, and listwise ranking. We apply our framework on two tasks: Sentence Ordering and Order Discrimination. Our framework outperforms various state-of-the-art methods on these tasks on a variety of evaluation metrics. We also show that it achieves better results when using pairwise and listwise ranking losses, rather than the pointwise ranking loss, which suggests that incorporating relative positions of two or more sentences in the loss function contributes to better learning.

Introduction

Sentence ordering—the task of arranging a set of shuffled sentences into a coherent paragraph—is a fundamental challenge in natural language processing. Coherence is essential for human readability and is a key requirement for downstream tasks such as multi-document summarization, question answering, and concept-to-text generation. Traditionally, this problem has been framed as a sequence-to-sequence task, often employing Pointer Networks to decode sentences one by one. However, these sequential models are computationally expensive and prone to error propagation, where a single mistake in the early stages of decoding can derail the entire sequence.

In this work, we propose RankTxNet, a novel framework that treats sentence ordering as a ranking problem. By globally scoring sentences based on their contextual relevance within a paragraph, our model enables parallel processing and achieves superior performance across diverse benchmarks.

The RankTxNet Architecture

The RankTxNet framework is composed of three primary modules: a Sentence Encoder, a Paragraph Encoder, and a scoring Decoder.

Overall architecture of the RankTxNet model, illustrating the flow from input sentences to final position scores.

1. Sentence Encoder (BERT)

Unlike previous approaches that relied on static word embeddings or LSTMs, RankTxNet utilizes a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to generate sentence-level representations. BERT's deep bidirectional context allows it to capture nuanced semantic information. A critical finding of our research is that fine-tuning BERT specifically for the sentence ordering task is essential; using "off-the-shelf" embeddings significantly degrades performance.

2. Paragraph Encoder (Transformer)

The paragraph encoder is a multi-layer Transformer network designed to model the inter-relationships between sentences. A key innovation here is the omission of positional encodings. By removing these encodings, the encoder becomes "order-invariant," meaning it treats the input as a set of sentences rather than a sequence. This ensures that the model's understanding of the paragraph's global context is not biased by the initial random shuffle of the input.

3. Decoder (FFNN)

The final stage is a simple Feed-Forward Neural Network (FFNN) that acts as a scoring decoder. It processes the contextualized sentence representations in parallel and assigns a relevance score to each. The final ordered sequence is obtained by sorting the sentences according to these scores. This parallel decoding is significantly more efficient than the step-by-step search required by Pointer Networks.

Detailed view of the RankTxNet components, including the BERT sentence encoder (left), the paragraph encoder (middle), and the internal Transformer block (right).

Training with Ranking Losses

RankTxNet is trained using three distinct ranking-based loss functions:

  • Pointwise Loss: Minimizes the difference between the predicted score and the actual position index.
  • Pairwise Loss: Ensures that for any two sentences, the one that should appear earlier receives a higher score.
  • Listwise Loss: Optimizes the probability of the entire sequence of scores matching the ground truth order.

Our experiments demonstrate that Pairwise and Listwise losses consistently outperform Pointwise loss, highlighting the importance of learning relative sentence positions for maintaining paragraph coherence.

Experimental Results

We evaluated RankTxNet on six benchmark datasets, including NIPS/AAN abstracts, SIND captions, and ROCStory commonsense stories. Performance was measured using Kendall’s tau ($\tau$), which quantifies the correlation between predicted and actual sequences.

  • State-of-the-Art Performance: RankTxNet achieved new SOTA results across all datasets. For instance, on the SIND dataset, it improved Kendall’s tau by 6.31% over previous models.
  • Coherence Benchmarking: In order discrimination tasks (e.g., the Accidents dataset), the model achieved 96.79% accuracy in distinguishing original paragraphs from shuffled versions.
  • Structural Accuracy: The model is particularly adept at identifying the first and last sentences, which are vital for establishing the narrative arc of a paragraph.

Conclusion

By reframing sentence ordering as a ranking task and leveraging the power of BERT and order-invariant Transformers, RankTxNet provides a robust and efficient solution for text coherence. Our results underscore the value of fine-tuning sentence representations and the superiority of relative ranking losses over absolute position prediction. Future work will explore incorporating visual context for multimodal datasets and scaling the architecture to handle even longer, more complex documents.

Reference: Kumar, P., Brahma, D., Karnick, H., & Rai, P. (2020). Deep Attentive Ranking Networks for Learning to Order Sentences. Proceedings of the AAAI Conference on Artificial Intelligence.

Summary updated

Abstract

We present an attention-based ranking framework for learning to order sentences given a paragraph. Our framework is built on a bidirectional sentence encoder and a self-attention based transformer network to obtain an input order invariant representation of paragraphs. Moreover, it allows seamless training using a variety of ranking based loss functions, such as pointwise, pairwise, and listwise ranking. We apply our framework on two tasks: Sentence Ordering and Order Discrimination. Our framework outperforms various state-of-the-art methods on these tasks on a variety of evaluation metrics. We also show that it achieves better results when using pairwise and listwise ranking losses, rather than the pointwise ranking loss, which suggests that incorporating relative positions of two or more sentences in the loss function contributes to better learning.
BibTeX
@inproceedings{kumar2020deep,
  author = {Pawan Kumar and Dhanajit Brahma and Harish Karnick and Piyush Rai},
  title = {{Deep Attentive Ranking Networks for Learning to Order Sentences}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=p1gQkjEvHYX},
}
Figure