RefSum: Refactoring Neural Summarization

Yixin Liu, Zi-Yi Dou, Pengfei Liu

Abstract

RefSum is a unified framework that treats both neural summarization and system combination as a "refactoring" task—specifically, a selection problem. By using a "pretrain-then-finetune" paradigm with shared parameters, it bridges the gap between base models and meta-learners. This approach addresses critical distribution shifts and parameter isolation in multi-stage summarization, achieving new state-of-the-art results on the CNN/DailyMail benchmark (46.18 ROUGE-1).

Introduction: The Two-Stage Paradigm

In modern Natural Language Processing (NLP), summarization often relies on a two-stage approach. First, a base stage (using models like BART or PEGASUS) generates several candidate summaries. Second, a meta stage (such as reranking or stacking) selects or combines these outputs to produce the final result. While effective, this setup suffers from three major limitations:

  1. Ad-hoc Methods: Most combiners are specialized for specific scenarios and lack generalizability.
  2. Base-Meta Learning Gap: Base and meta-models rarely share parameters, meaning the meta-model cannot leverage the base model's internal understanding of the source document.
  3. Train-Test Distribution Gap: Meta-models are often trained on high-quality "hypotheses" close to the ground truth. At test time, they encounter lower-quality, "messier" outputs from base models, leading to poor selection performance.
The RefSum framework identifies and addresses the structural and distributional gaps between base and meta stages.

Methods: Summarization as Selection

The authors propose Refactor, a model that unifies the base and meta stages under a single selection objective. The core insight is that whether a model is picking sentences from a source document (extractive summarization) or picking the best output from multiple AI models (system combination), the underlying criteria for a "good summary" remain the same.

The Architecture

Refactor utilizes a BERT-based backbone to calculate contextualized similarity between a document and candidate summaries. It incorporates an Importance Weighting module—a two-layer Transformer that assigns weights to document tokens—ensuring the model focuses on the most salient parts of the text. The system is trained using a margin-based ranking loss, penalizing the model when it fails to rank summaries with higher ROUGE scores above those with lower scores.

The "Pretrain-then-Finetune" Paradigm

To solve the distribution gap, the authors introduce a two-step training process:

  1. Pre-training: The model learns to select the best combination of actual sentences from the source document. This provides a massive, diverse set of candidates that "toughens" the model.
  2. Fine-tuning: The model is then tuned on the actual outputs of state-of-the-art abstractive systems. Because it was pre-trained on diverse extractive combinations, it handles the varied quality of test-time system outputs with significantly higher robustness.

Key Results

The authors evaluated RefSum across four diverse benchmarks: CNN/DailyMail (CNNDM), XSum, PubMed, and WikiHow. To test generalization, they applied RefSum to 19 different top-performing systems on the CNNDM leaderboard.

Performance comparison on CNN/DailyMail, showing RefSum (Fine-tuned) achieving state-of-the-art results across ROUGE metrics.

RefSum achieved a record-breaking 46.18 ROUGE-1 on CNN/DailyMail. Beyond raw scores, the model demonstrated "Universal Improvement," consistently boosting the performance of existing systems regardless of whether the base model was extractive or abstractive. It also proved effective at Sentence-Level Fusion, assembling "Frankenstein" summaries from different systems that outperformed any individual input model.

Conclusion and Future Work

RefSum demonstrates that unifying base and meta-learning through parameter sharing and strategic pre-training can overcome long-standing hurdles in multi-stage summarization. While the model is currently a "selector" bounded by the quality of its candidates, it provides an off-the-shelf tool for immediate performance gains. Future research may move toward more fine-grained "refactoring," where models actively edit and merge candidates at a sub-sentence level rather than just selecting them.

Summary updated

Abstract

Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement. Our system can be directly used by other researchers as an off-the-shelf tool to achieve further performance improvements. We open-source all the code and provide a convenient interface to use it: https://github.com/yixinL7/Refactoring-Summarization. We have also made the demo of this work available at: http://explainaboard.nlpedia.ai/leaderboard/task-summ/index.php.
BibTeX
@inproceedings{liu2021refsum,
  author = {Yixin Liu and Zi-Yi Dou and Pengfei Liu},
  title = {{RefSum: Refactoring Neural Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.113},
}
Figure