Balancing Lexical and Semantic Quality in Abstractive Summarization

Jeewoo Sul, Yong Suk Choi

Introduction

Modern abstractive summarization typically relies on sequence-to-sequence (Seq2Seq) architectures trained via Maximum Likelihood Estimation. While effective, this approach is prone to exposure bias, where models struggle to recover from small errors during inference because they were only exposed to ground-truth data during training. To mitigate this, researchers often employ a two-stage process: a generator produces multiple candidate summaries, and a re-ranker selects the optimal one.

However, most re-rankers fall into the "ROUGE trap," prioritizing lexical overlap (word-for-word matching) over semantic fidelity. Empirical evidence suggests a significant gap between these metrics; in approximately 52% of cases, the candidate with the highest ROUGE score is not the most semantically accurate as measured by BERTScore. This discrepancy highlights the need for a re-ranking mechanism that balances both lexical and semantic quality.

Figure 1: The distribution of the ranking of the highest BERTScore summary among candidates ranked by ROUGE, illustrating that the top ROUGE candidate is often not the most semantically accurate.

The BalSum Framework

The authors propose BalSum, a novel re-ranking framework designed to bridge the lexical-semantic gap. Built on a RoBERTa bi-encoder architecture, BalSum introduces three core innovations to improve summary selection:

  1. Granular Sentence-Level Encoding: Instead of processing documents as monolithic blocks, BalSum inserts [CLS] tokens before every sentence. This allows the model to capture fine-grained semantic units, which are then aggregated to compute a comprehensive similarity score.
  2. Refined Ranking Loss: BalSum’s loss function explicitly incorporates the numerical difference in ROUGE scores between candidates. This ensures the model maintains lexical quality where appropriate rather than treating all ranking margins as equal.
  3. Contrastive Loss with Instance Weighting: To address "false positives"—summaries with high lexical overlap but poor semantic meaning—BalSum uses a semantic similarity threshold ($\phi$). Candidates falling below this threshold are treated as noise, and their influence on training is reduced through instance weighting.
Figure 2: The BalSum architecture, utilizing summary and document encoders to produce a total similarity score based on aggregated sentence-level representations.

Experimental Results

BalSum was evaluated on two major benchmarks: CNN/DailyMail (multi-sentence news summaries) and XSum (extreme single-sentence summaries).

  • State-of-the-Art Performance: On the CNN/DailyMail dataset, BalSum achieved a BERTScore of 89.67, surpassing previous leading models such as SimCLS and BRIO.
  • Generator Agnosticism: The framework demonstrated consistent performance gains across different base generators, including BART and PEGASUS.
  • Ablation Studies: Results confirmed that the instance weighting threshold ($\phi$) is critical. Increasing the threshold—thereby being more selective about semantic quality—resulted in measurable improvements in the final summary quality.

Conclusion and Future Work

BalSum effectively addresses the ROUGE trap by integrating semantic awareness into the re-ranking process. While the model shows significant improvements on long-form datasets like CNN/DailyMail, its impact on extremely short summaries (XSum) is more modest, likely due to the limited room for capturing diverse semantic units in single-sentence outputs.

Future research will explore methods to extract more detailed features from input text to enhance short-form summarization and investigate diverse decoding strategies to provide re-rankers with a higher-quality pool of initial candidates.

Summary updated

Abstract

An important problem of the sequence-to-sequence neural models widely used in abstractive summarization is exposure bias. To alleviate this problem, re-ranking systems have been applied in recent years. Despite some performance improvements, this approach remains underexplored. Previous works have mostly specified the rank through the ROUGE score and aligned candidate summaries, but there can be quite a large gap between the lexical overlap metric and semantic similarity. In this paper, we propose a novel training method in which a re-ranker balances the lexical and semantic quality. We further newly define false positives in ranking and present a strategy to reduce their influence. Experiments on the CNN/DailyMail and XSum datasets show that our method can estimate the meaning of summaries without seriously degrading the lexical aspect. More specifically, it achieves an 89.67 BERTScore on the CNN/DailyMail dataset, reaching new state-of-the-art performance. Our code is publicly available at https://github.com/jeewoo1025/BalSum.
BibTeX
@inproceedings{sul2023balancing,
  author = {Jeewoo Sul and Yong Suk Choi},
  title = {{Balancing Lexical and Semantic Quality in Abstractive Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=yGY0HSIwCsz},
}
Figure