Introduction
Abstractive summarization aims to generate concise, paraphrased versions of source documents. However, the performance of these models is heavily influenced by the "extractivity" of their training data—the degree to which target summaries overlap word-for-word with the source. This paper explores how high extractivity can lead to "lazy" copying behaviors and proposes a mitigation strategy using Integer Linear Programming (ILP) to guide the model toward salient content.
The Extractivity Trade-off
The researchers found that model performance does not scale linearly with dataset extractivity. While a moderate amount of extractive overlap helps models ground their summaries in factual content, excessive extractivity leads to a performance "nosedive."
Continuity and Salience
To quantify this behavior, the authors introduced two new metrics:
- Continuity: The proportion of the summary composed of long, continuous sequences from the source.
- Continuity Salience: A measure of the importance of these copied sequences relative to the ground-truth summary.
As training sets become more extractive, models exhibit higher Continuity but lower Continuity Salience. In other words, they learn to copy longer spans of text, but the quality and relevance of the copied information decrease as the model "overfits" to copying behavior.
Methodology: ILP-Guided Copying
To address the issue of "lazy" copying, the authors proposed a training strategy that uses Integer Linear Programming (ILP) to identify optimal copy labels.
The ILP formulation identifies the minimum set of source sentences required to cover the extractive portions of a summary. This process distinguishes between "Gold" labels (tokens in optimal sentences) and "Silver" labels (other matches). An auxiliary KL-Divergence loss then penalizes the model if its internal copy mechanism deviates from these importance-based labels, forcing it to focus on salient information rather than just long strings.
Experimental Results
The study evaluated BART and PEGASUS models on the CNN/Daily Mail (news) and arXiv/PubMed (scientific) datasets.
The Inverted U Relationship
The results across different extractivity tiers (T1: low, T2: medium, T3: high) demonstrate that performance peaks at moderate extractivity levels before declining significantly in the high-extractivity tier.
Mitigation Performance
The proposed BART + Copy Labels method consistently outperformed standard baselines. For instance, on CNN/DM, it achieved a ROUGE-1 score of 45.21, surpassing the standard BART baseline of 44.12. Similar gains were observed on the arXiv/PubMed dataset, where it reached a ROUGE-1 of 45.97. Qualitatively, the method reduced blind copying (lower Continuity) while ensuring that the segments it did copy were more relevant (higher Continuity Salience).
Conclusion and Future Directions
This research highlights a critical pitfall in training abstractive models on highly extractive data. By introducing ILP-based copy labels, the authors provide a way to maintain abstractive quality even when the data encourages verbatim copying. Future work may involve moving from sentence-level to token-level salience identification and exploring semantic-based metrics to better capture the essence of summaries beyond simple string overlap.
Summary updated