Abstract
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei, Ming Zhou. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
Introduction
State-of-the-art summarization models typically rely on large-scale supervised datasets, where documents are paired with human-written summaries. However, acquiring such labels is labor-intensive and expensive. This has led to a growing interest in unsupervised extractive summarization, which aims to identify the most salient sentences in a document without any labeled examples.
Traditional unsupervised methods often utilize graph-based algorithms like TextRank to rank sentences based on similarity. While effective, these methods frequently suffer from "lead bias"—the tendency to prioritize the beginning of a document—and may fail to capture complex semantic relationships. This paper introduces STAS (Sentence-level Transformer based Attentive Summarization), a novel framework that leverages hierarchical Transformers and self-supervised pre-training to identify document salience.
Methodology: The STAS Framework
STAS utilizes a hierarchical Transformer architecture designed to capture both local word-level context and global document-level structure.
Hierarchical Architecture
The model consists of two primary levels:
- Token-level Transformer: This level processes individual tokens to generate dense vector representations for each sentence.
- Sentence-level Transformer: This level takes the sentence representations as input and models the inter-sentence relationships across the entire document.
Self-Supervised Pre-training
To train the model without human labels, the authors propose two novel pre-training objectives:
- Masked Sentences Prediction (MSP): Similar to the masked language modeling used in BERT, the model masks a percentage of sentences and attempts to reconstruct them based on the surrounding context. This encourages the model to learn the semantic probability of a sentence within a document.
- Sentence Shuffling (SS): The model is presented with a document where the sentence order has been randomized. It must then predict the original sequence. This task forces the model to understand the logical flow and relative importance of sentences, effectively mitigating lead bias by focusing on content rather than position.
Ranking via Attention Centrality
Once pre-trained, STAS ranks sentences using a combination of their reconstruction probability (from MSP) and Attention Centrality. The latter is derived from the self-attention matrix of the sentence-level Transformer. If many other sentences in the document "attend" to a specific sentence with high weights, that sentence is considered highly salient.
Experimental Evaluation
The model was evaluated on two benchmark datasets: CNN/DailyMail (news articles with bullet-point highlights) and the New York Times (NYT) dataset (articles with summaries by library scientists). Performance was measured using ROUGE scores, comparing STAS against traditional unsupervised methods (TextRank), recent unsupervised models (PACSUM), and established supervised baselines.
Key Results
STAS achieved state-of-the-art performance for unsupervised extractive summarization across both datasets.
As shown in the results table, STAS significantly outperforms traditional graph-based methods like TextRank and surpasses previous unsupervised leaders like PACSUM. On the CNN/DM dataset, STAS achieved a ROUGE-L score of 37.21, while on the NYT dataset, it reached 37.57, notably outperforming the Lead-3 baseline (32.00).
Furthermore, qualitative analysis indicates that STAS selects sentences more uniformly from across the document compared to previous models, demonstrating its ability to identify important information regardless of its position.
Conclusion
STAS demonstrates that the internal attention mechanisms of a pre-trained hierarchical Transformer can serve as a powerful proxy for human-perceived salience. By combining masked sentence prediction with sentence shuffling, the model learns to prioritize content over simple positional heuristics. Future work will explore extending this hierarchical attention-based approach to unsupervised abstractive summarization.
Summary updated