Abstract
Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. In this work, we find that transformer attentions can be used to rank sentences for unsupervised extractive summarization. Specifically, we first pre-train a hierarchical transformer model using unlabeled documents only. Then we propose a method to rank sentences using sentence-level self-attentions and pre-training objectives. Experiments on CNN/DailyMail and New York Times datasets show our model achieves state-of-the-art performance on unsupervised summarization. We also find in experiments that our model is less dependent on sentence positions. When using a linear combination of our model and a recent unsupervised model explicitly modeling sentence positions, we obtain even better results.
Introduction
Most state-of-the-art summarization models are supervised, requiring massive datasets of document-summary pairs. These are expensive to produce and do not scale well across different languages or domains. Unsupervised extractive summarization—selecting the most important sentences from a text without labeled data—has traditionally relied on graph-based algorithms like TextRank. These methods treat sentences as nodes and use similarity scores to find "central" sentences. However, they often suffer from "lead bias" (simply picking the first few sentences) or fail to capture deep contextual interactions. This paper introduces STAS (Sentence-level Transformer based Attentive Summarization), leveraging Transformer architectures to create a sophisticated, content-aware ranking system.
Methodology
The STAS model utilizes a hierarchical Transformer architecture to capture document structure at multiple levels.
As shown in Figure 1, the architecture consists of two main components:
- Hierarchical Encoding: A token-level Transformer (initialized with RoBERTa) encodes words into sentence representations. Subsequently, a sentence-level Transformer learns the interactions between these sentences.
- Pre-training Tasks: To understand document structure without labels, the model is pre-trained using two objectives:
- Masked Sentences Prediction (MSP): The model hides 15% of sentences and attempts to reconstruct them based on surrounding context.
- Sentence Shuffling (SS): The model receives a document with shuffled sentences and must predict their original order, forcing it to learn content-based importance rather than relying solely on position.
Ranking for Summarization
At test time, STAS ranks sentences using two criteria derived from the pre-trained model:
- Probability Score ($r_i$): Measures how likely a sentence is given the rest of the document context, calculated via the MSP objective.
- Attention Score ($r'_i$): Measures how much other sentences "attend" to a specific sentence in the sentence-level Transformer. A sentence that many others "look at" to understand context is likely salient.
Results and Discussion
The model was evaluated on the CNN/DailyMail (CNN/DM) and New York Times (NYT) benchmarks. STAS achieved state-of-the-art performance for unsupervised summarization on both datasets.
As seen in the results table, STAS (MSP+SS) achieves competitive ROUGE scores. On the NYT dataset, the unsupervised STAS combination even outperformed some older supervised models like REFRESH. On CNN/DM, STAS achieved a ROUGE-L score of 37.21, significantly outperforming the popular TextRank (27.40) and the previous SOTA unsupervised model, PACSUM (36.90).
One of the most significant findings is STAS's ability to break "lead bias." By using the Sentence Shuffling task, the model learns to identify importance based on content rather than location. Analysis showed that STAS's extracted sentences are more smoothly distributed across the document compared to other models that cluster heavily at the beginning.
Conclusion
STAS demonstrates that internal attention scores and sentence probabilities from a pre-trained hierarchical Transformer can effectively rank sentence importance without human-written labels. Future work may involve applying these hierarchical pre-training techniques to unsupervised abstractive summarization to allow for rewriting and synthesis of information.
Summary updated