Introduction
Modern abstractive summarization typically relies on sequence-to-sequence (Seq2Seq) architectures trained via Maximum Likelihood Estimation. While effective, this approach is prone to exposure bias, where models struggle to recover from small errors during inference because they were only exposed to ground-truth data during training. To mitigate this, researchers often employ a two-stage process: a generator produces multiple candidate summaries, and a re-ranker selects the optimal one.
However, most re-rankers fall into the "ROUGE trap," prioritizing lexical overlap (word-for-word matching) over semantic fidelity. Empirical evidence suggests a significant gap between these metrics; in approximately 52% of cases, the candidate with the highest ROUGE score is not the most semantically accurate as measured by BERTScore. This discrepancy highlights the need for a re-ranking mechanism that balances both lexical and semantic quality.
The BalSum Framework
The authors propose BalSum, a novel re-ranking framework designed to bridge the lexical-semantic gap. Built on a RoBERTa bi-encoder architecture, BalSum introduces three core innovations to improve summary selection:
- Granular Sentence-Level Encoding: Instead of processing documents as monolithic blocks, BalSum inserts
[CLS]tokens before every sentence. This allows the model to capture fine-grained semantic units, which are then aggregated to compute a comprehensive similarity score. - Refined Ranking Loss: BalSum’s loss function explicitly incorporates the numerical difference in ROUGE scores between candidates. This ensures the model maintains lexical quality where appropriate rather than treating all ranking margins as equal.
- Contrastive Loss with Instance Weighting: To address "false positives"—summaries with high lexical overlap but poor semantic meaning—BalSum uses a semantic similarity threshold ($\phi$). Candidates falling below this threshold are treated as noise, and their influence on training is reduced through instance weighting.
Experimental Results
BalSum was evaluated on two major benchmarks: CNN/DailyMail (multi-sentence news summaries) and XSum (extreme single-sentence summaries).
- State-of-the-Art Performance: On the CNN/DailyMail dataset, BalSum achieved a BERTScore of 89.67, surpassing previous leading models such as SimCLS and BRIO.
- Generator Agnosticism: The framework demonstrated consistent performance gains across different base generators, including BART and PEGASUS.
- Ablation Studies: Results confirmed that the instance weighting threshold ($\phi$) is critical. Increasing the threshold—thereby being more selective about semantic quality—resulted in measurable improvements in the final summary quality.
Conclusion and Future Work
BalSum effectively addresses the ROUGE trap by integrating semantic awareness into the re-ranking process. While the model shows significant improvements on long-form datasets like CNN/DailyMail, its impact on extremely short summaries (XSum) is more modest, likely due to the limited room for capturing diverse semantic units in single-sentence outputs.
Future research will explore methods to extract more detailed features from input text to enhance short-form summarization and investigate diverse decoding strategies to provide re-rankers with a higher-quality pool of initial candidates.
Summary updated