RefSum: Refactoring Neural Summarization

Yixin Liu, Zi-Yi Dou, Pengfei Liu

Abstract: Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement.

Introduction

In neural text summarization, researchers have flexible choices in model architectures and decoding strategies (like beam search). Because of these different inductive biases, different models generate diverse summaries for the same text. To take advantage of this, researchers sometimes use two-stage learning: a base stage generates multiple candidate summaries, and a meta stage aggregates or reranks them (often called "stacking" or "reranking").

However, existing two-stage methods suffer from three major limitations: ad-hoc designs that don't generalize to multiple systems, a learning gap where parameters aren't shared between stages, and a distribution gap between the high-quality training candidates and imperfect test-time outputs.

Methods

The authors propose Refactor, a framework that unifies the base and meta stages by framing both as a sentence selection problem. Whether the model is picking sentences directly from the source document (base) or picking from a list of generated summaries (meta), the criteria for what makes a "good" summary remain the same.

The Refactor framework unifies base and meta stages to bridge learning and distribution gaps.

Refactor uses a Pre-train then Fine-tune paradigm to solve the distribution gaps:

  • Pre-trained Refactor: The model is first trained to score and select candidate summaries generated by heuristically combining sentences directly from the source document. It uses a BERT-based contextualized similarity function to compare the document and the candidate summary, optimized via a ranking loss based on ROUGE scores. This exposes the model to a massive, diverse set of candidate qualities.
  • Fine-tuned Refactor: To adapt to the specific quirks of the base models, Refactor is then fine-tuned on the actual outputs generated by the base systems it will be evaluating at test time.

Datasets

The approach was evaluated on four standard summarization datasets:

  • CNN/DailyMail (CNNDM): News articles with bullet-point highlights.
  • XSum: BBC news articles with highly abstractive, one-sentence summaries.
  • PubMed: Long scientific papers.
  • WikiHow: How-to articles with step-by-step summaries.

Key Results and Insights

Refactor achieves significant performance gains by addressing the limitations of traditional reranking. A key insight is that standard beam search often leaves performance on the table; reranking the top-4 beam search outputs of a single model yields significant gains. Furthermore, pre-training on heuristic combinations of source sentences allows the model to see a much wider distribution of summary qualities, preventing overfitting to the narrow distribution of a base model's training outputs.

Performance comparison of Refactor against various baselines on BART and GSum systems.

As shown in the results, Refactor achieved a new state-of-the-art on the CNN/DailyMail dataset with a 46.18 ROUGE-1 score by reranking the outputs of the GSum model. Even for a standalone BART model, Refactor improved the ROUGE-1 score from 44.26 to 45.15. Additionally, the pre-trained Refactor model demonstrated strong zero-shot generalization, improving the performance of 19 different top-performing systems without system-specific fine-tuning.

Conclusion

While Refactor is highly effective, its gains are most pronounced when base systems have similar performance but different error profiles. If one system is overwhelmingly stronger, the meta-model finds less complementary information to extract. This work suggests a paradigm shift toward stage-wise learning, where different inductive biases are injected at different stages to fully utilize the complementarity of diverse architectures and decoding strategies.

Summary updated

Abstract

Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement. Our system can be directly used by other researchers as an off-the-shelf tool to achieve further performance improvements. We open-source all the code and provide a convenient interface to use it: https://github.com/yixinL7/Refactoring-Summarization. We have also made the demo of this work available at: http://explainaboard.nlpedia.ai/leaderboard/task-summ/index.php.
BibTeX
@inproceedings{liu2021refsum,
  author = {Yixin Liu and Zi-Yi Dou and Pengfei Liu},
  title = {{RefSum: Refactoring Neural Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.113},
}
Figure