Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers

Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei, Ming Zhou

Abstract

Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei, Ming Zhou. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction

State-of-the-art summarization models typically rely on large-scale supervised datasets, where documents are paired with human-written summaries. However, acquiring such labels is labor-intensive and expensive. This has led to a growing interest in unsupervised extractive summarization, which aims to identify the most salient sentences in a document without any labeled examples.

Traditional unsupervised methods often utilize graph-based algorithms like TextRank to rank sentences based on similarity. While effective, these methods frequently suffer from "lead bias"—the tendency to prioritize the beginning of a document—and may fail to capture complex semantic relationships. This paper introduces STAS (Sentence-level Transformer based Attentive Summarization), a novel framework that leverages hierarchical Transformers and self-supervised pre-training to identify document salience.

Methodology: The STAS Framework

STAS utilizes a hierarchical Transformer architecture designed to capture both local word-level context and global document-level structure.

The hierarchical architecture of STAS, featuring token-level and sentence-level Transformers.

Hierarchical Architecture

The model consists of two primary levels:

  1. Token-level Transformer: This level processes individual tokens to generate dense vector representations for each sentence.
  2. Sentence-level Transformer: This level takes the sentence representations as input and models the inter-sentence relationships across the entire document.

Self-Supervised Pre-training

To train the model without human labels, the authors propose two novel pre-training objectives:

  • Masked Sentences Prediction (MSP): Similar to the masked language modeling used in BERT, the model masks a percentage of sentences and attempts to reconstruct them based on the surrounding context. This encourages the model to learn the semantic probability of a sentence within a document.
  • Sentence Shuffling (SS): The model is presented with a document where the sentence order has been randomized. It must then predict the original sequence. This task forces the model to understand the logical flow and relative importance of sentences, effectively mitigating lead bias by focusing on content rather than position.

Ranking via Attention Centrality

Once pre-trained, STAS ranks sentences using a combination of their reconstruction probability (from MSP) and Attention Centrality. The latter is derived from the self-attention matrix of the sentence-level Transformer. If many other sentences in the document "attend" to a specific sentence with high weights, that sentence is considered highly salient.

Experimental Evaluation

The model was evaluated on two benchmark datasets: CNN/DailyMail (news articles with bullet-point highlights) and the New York Times (NYT) dataset (articles with summaries by library scientists). Performance was measured using ROUGE scores, comparing STAS against traditional unsupervised methods (TextRank), recent unsupervised models (PACSUM), and established supervised baselines.

Key Results

STAS achieved state-of-the-art performance for unsupervised extractive summarization across both datasets.

Performance comparison of STAS against various unsupervised and supervised baselines on CNN/DM and NYT datasets.

As shown in the results table, STAS significantly outperforms traditional graph-based methods like TextRank and surpasses previous unsupervised leaders like PACSUM. On the CNN/DM dataset, STAS achieved a ROUGE-L score of 37.21, while on the NYT dataset, it reached 37.57, notably outperforming the Lead-3 baseline (32.00).

Furthermore, qualitative analysis indicates that STAS selects sentences more uniformly from across the document compared to previous models, demonstrating its ability to identify important information regardless of its position.

Conclusion

STAS demonstrates that the internal attention mechanisms of a pre-trained hierarchical Transformer can serve as a powerful proxy for human-perceived salience. By combining masked sentence prediction with sentence shuffling, the model learns to prioritize content over simple positional heuristics. Future work will explore extending this hierarchical attention-based approach to unsupervised abstractive summarization.

Summary updated

BibTeX
@inproceedings{xu2020unsupervised,
  author = {Shusheng Xu and Xingxing Zhang and Yi Wu and Furu Wei and Ming Zhou},
  title = {{Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=JOSNfmhgscD},
}
Figure