Abstract: Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement.
Introduction
In neural text summarization, researchers have flexible choices in model architectures and decoding strategies (like beam search). Because of these different inductive biases, different models generate diverse summaries for the same text. To take advantage of this, researchers sometimes use two-stage learning: a base stage generates multiple candidate summaries, and a meta stage aggregates or reranks them (often called "stacking" or "reranking").
However, existing two-stage methods suffer from three major limitations: ad-hoc designs that don't generalize to multiple systems, a learning gap where parameters aren't shared between stages, and a distribution gap between the high-quality training candidates and imperfect test-time outputs.
Methods
The authors propose Refactor, a framework that unifies the base and meta stages by framing both as a sentence selection problem. Whether the model is picking sentences directly from the source document (base) or picking from a list of generated summaries (meta), the criteria for what makes a "good" summary remain the same.
Refactor uses a Pre-train then Fine-tune paradigm to solve the distribution gaps:
- Pre-trained Refactor: The model is first trained to score and select candidate summaries generated by heuristically combining sentences directly from the source document. It uses a BERT-based contextualized similarity function to compare the document and the candidate summary, optimized via a ranking loss based on ROUGE scores. This exposes the model to a massive, diverse set of candidate qualities.
- Fine-tuned Refactor: To adapt to the specific quirks of the base models, Refactor is then fine-tuned on the actual outputs generated by the base systems it will be evaluating at test time.
Datasets
The approach was evaluated on four standard summarization datasets:
- CNN/DailyMail (CNNDM): News articles with bullet-point highlights.
- XSum: BBC news articles with highly abstractive, one-sentence summaries.
- PubMed: Long scientific papers.
- WikiHow: How-to articles with step-by-step summaries.
Key Results and Insights
Refactor achieves significant performance gains by addressing the limitations of traditional reranking. A key insight is that standard beam search often leaves performance on the table; reranking the top-4 beam search outputs of a single model yields significant gains. Furthermore, pre-training on heuristic combinations of source sentences allows the model to see a much wider distribution of summary qualities, preventing overfitting to the narrow distribution of a base model's training outputs.
As shown in the results, Refactor achieved a new state-of-the-art on the CNN/DailyMail dataset with a 46.18 ROUGE-1 score by reranking the outputs of the GSum model. Even for a standalone BART model, Refactor improved the ROUGE-1 score from 44.26 to 45.15. Additionally, the pre-trained Refactor model demonstrated strong zero-shot generalization, improving the performance of 19 different top-performing systems without system-specific fine-tuning.
Conclusion
While Refactor is highly effective, its gains are most pronounced when base systems have similar performance but different error profiles. If one system is overwhelmingly stronger, the meta-model finds less complementary information to extract. This work suggests a paradigm shift toward stage-wise learning, where different inductive biases are injected at different stages to fully utilize the complementarity of diverse architectures and decoding strategies.
Summary updated