Abstract
Abstractive document summarization is usually modeled as a sequence-to-sequence (Seq2Seq) learning problem. Unfortunately, training large Seq2Seq based summarization models on limited supervised summarization data is challenging. This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text. The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document. These objectives include sentence reordering, next sentence generation, and masked document generation, which have close relations with the abstractive document summarization task. Experiments on two benchmark summarization datasets (i.e., CNN/DailyMail and New York Times) show that all three objectives can improve performance upon baselines. Compared to models pre-trained on large-scale data (more than 160GB), our method, with only 19GB text for pre-training, achieves comparable results, which demonstrates its effectiveness.
Introduction
Abstractive summarization is a complex Sequence-to-Sequence (Seq2Seq) task requiring a model to read a document and generate a concise, paraphrased summary. Unlike extractive methods that copy sentences directly, abstractive models must grasp global context and reorganize information. However, the field faces significant hurdles: high-quality human-written summaries are scarce, and popular pre-training methods like BERT often prioritize the encoder over the generative decoder. Furthermore, human summaries frequently reorder content—a structural nuance found in roughly 40% of standard training data—which many models fail to explicitly address.
Methodology: The STEP Framework
To address these challenges, the authors introduce STEP (Sequence-to-Sequence Transformer Pre-training). The framework utilizes a standard Seq2Seq Transformer, initialized with a RoBERTa-Large encoder and a 6-layer decoder. The core innovation lies in three self-supervised pre-training objectives designed to teach the model how to "reinstate" original documents from corrupted versions:
- Sentence Reordering (SR): The model receives a document with shuffled sentences and must restore the original, coherent order. This mimics the human process of reorganizing information during summarization.
- Next Sentence Generation (NSG): Given the first segment of a document, the model predicts the subsequent segment, learning to anticipate logical flow.
- Masked Document Generation (MDG): The model reinstates a document where spans of text have been masked or replaced, similar to BERT's fill-in-the-blank task but applied to document-level generation.
Datasets and Evaluation
The model was pre-trained on GIGA-CM, a 19GB unlabeled corpus of 6.5 million documents. Evaluation was conducted on two primary benchmarks:
- CNN/DailyMail (CNNDM): A standard long-form news summarization dataset.
- New York Times (NYT): A dataset featuring multi-sentence summaries written by experts.
Key Results and Insights
Despite being trained on significantly less data (19GB) compared to massive models like BART (160GB) or T5 (750GB), STEP achieved state-of-the-art or comparable results. On the CNNDM benchmark, STEP reached a ROUGE-L score of 41.24, outperforming UniLM and matching much larger models.
Key insights from the study include:
- Task Alignment: The specific design of pre-training objectives (like SR) provides a superior inductive bias for summarization compared to sheer data volume.
- Decoder Pre-training: Pre-training the encoder and decoder together on generative tasks is essential for high-performance abstractive summarization.
- Human Preference: In blind human evaluations, STEP was ranked as the best output in 23% of cases, significantly higher than competitors like BERTAbs and UniLM.
Limitations and Future Work
STEP is currently limited by the standard Transformer input constraint of 512 tokens, which may lead to information loss in very long documents. Additionally, the Sentence Reordering objective is less effective for single-sentence summary datasets like XSum. Future research will focus on extending the model's memory for longer documents and developing objectives suited for extreme summarization tasks.
Summary updated