Abstract
RefSum is a unified framework that treats both neural summarization and system combination as a "refactoring" task—specifically, a selection problem. By using a "pretrain-then-finetune" paradigm with shared parameters, it bridges the gap between base models and meta-learners. This approach addresses critical distribution shifts and parameter isolation in multi-stage summarization, achieving new state-of-the-art results on the CNN/DailyMail benchmark (46.18 ROUGE-1).
Introduction: The Two-Stage Paradigm
In modern Natural Language Processing (NLP), summarization often relies on a two-stage approach. First, a base stage (using models like BART or PEGASUS) generates several candidate summaries. Second, a meta stage (such as reranking or stacking) selects or combines these outputs to produce the final result. While effective, this setup suffers from three major limitations:
- Ad-hoc Methods: Most combiners are specialized for specific scenarios and lack generalizability.
- Base-Meta Learning Gap: Base and meta-models rarely share parameters, meaning the meta-model cannot leverage the base model's internal understanding of the source document.
- Train-Test Distribution Gap: Meta-models are often trained on high-quality "hypotheses" close to the ground truth. At test time, they encounter lower-quality, "messier" outputs from base models, leading to poor selection performance.
Methods: Summarization as Selection
The authors propose Refactor, a model that unifies the base and meta stages under a single selection objective. The core insight is that whether a model is picking sentences from a source document (extractive summarization) or picking the best output from multiple AI models (system combination), the underlying criteria for a "good summary" remain the same.
The Architecture
Refactor utilizes a BERT-based backbone to calculate contextualized similarity between a document and candidate summaries. It incorporates an Importance Weighting module—a two-layer Transformer that assigns weights to document tokens—ensuring the model focuses on the most salient parts of the text. The system is trained using a margin-based ranking loss, penalizing the model when it fails to rank summaries with higher ROUGE scores above those with lower scores.
The "Pretrain-then-Finetune" Paradigm
To solve the distribution gap, the authors introduce a two-step training process:
- Pre-training: The model learns to select the best combination of actual sentences from the source document. This provides a massive, diverse set of candidates that "toughens" the model.
- Fine-tuning: The model is then tuned on the actual outputs of state-of-the-art abstractive systems. Because it was pre-trained on diverse extractive combinations, it handles the varied quality of test-time system outputs with significantly higher robustness.
Key Results
The authors evaluated RefSum across four diverse benchmarks: CNN/DailyMail (CNNDM), XSum, PubMed, and WikiHow. To test generalization, they applied RefSum to 19 different top-performing systems on the CNNDM leaderboard.
RefSum achieved a record-breaking 46.18 ROUGE-1 on CNN/DailyMail. Beyond raw scores, the model demonstrated "Universal Improvement," consistently boosting the performance of existing systems regardless of whether the base model was extractive or abstractive. It also proved effective at Sentence-Level Fusion, assembling "Frankenstein" summaries from different systems that outperformed any individual input model.
Conclusion and Future Work
RefSum demonstrates that unifying base and meta-learning through parameter sharing and strategic pre-training can overcome long-standing hurdles in multi-stage summarization. While the model is currently a "selector" bounded by the quality of its candidates, it provides an off-the-shelf tool for immediate performance gains. Future research may move toward more fine-grained "refactoring," where models actively edit and merge candidates at a sub-sentence level rather than just selecting them.
Summary updated