Detection and Mitigation of the Negative Impact of Dataset Extractivity on Abstractive Summarization

Yubin Ge, Sullam Jeoung, Ly Dinh, Jana Diesner

Introduction

Abstractive summarization aims to generate concise, paraphrased versions of source documents. However, the performance of these models is heavily influenced by the "extractivity" of their training data—the degree to which target summaries overlap word-for-word with the source. This paper explores how high extractivity can lead to "lazy" copying behaviors and proposes a mitigation strategy using Integer Linear Programming (ILP) to guide the model toward salient content.

The Extractivity Trade-off

The researchers found that model performance does not scale linearly with dataset extractivity. While a moderate amount of extractive overlap helps models ground their summaries in factual content, excessive extractivity leads to a performance "nosedive."

Continuity and Salience

To quantify this behavior, the authors introduced two new metrics:

  • Continuity: The proportion of the summary composed of long, continuous sequences from the source.
  • Continuity Salience: A measure of the importance of these copied sequences relative to the ground-truth summary.

As training sets become more extractive, models exhibit higher Continuity but lower Continuity Salience. In other words, they learn to copy longer spans of text, but the quality and relevance of the copied information decrease as the model "overfits" to copying behavior.

Methodology: ILP-Guided Copying

To address the issue of "lazy" copying, the authors proposed a training strategy that uses Integer Linear Programming (ILP) to identify optimal copy labels.

Overview of the source-target overlap and the ILP optimization process used to identify salient source sentences for copy-guidance.

The ILP formulation identifies the minimum set of source sentences required to cover the extractive portions of a summary. This process distinguishes between "Gold" labels (tokens in optimal sentences) and "Silver" labels (other matches). An auxiliary KL-Divergence loss then penalizes the model if its internal copy mechanism deviates from these importance-based labels, forcing it to focus on salient information rather than just long strings.

Experimental Results

The study evaluated BART and PEGASUS models on the CNN/Daily Mail (news) and arXiv/PubMed (scientific) datasets.

The Inverted U Relationship

The results across different extractivity tiers (T1: low, T2: medium, T3: high) demonstrate that performance peaks at moderate extractivity levels before declining significantly in the high-extractivity tier.

Performance metrics for BART and PEGASUS across different dataset extractivity (DE) levels, illustrating the performance drop at high extractivity (T3).

Mitigation Performance

The proposed BART + Copy Labels method consistently outperformed standard baselines. For instance, on CNN/DM, it achieved a ROUGE-1 score of 45.21, surpassing the standard BART baseline of 44.12. Similar gains were observed on the arXiv/PubMed dataset, where it reached a ROUGE-1 of 45.97. Qualitatively, the method reduced blind copying (lower Continuity) while ensuring that the segments it did copy were more relevant (higher Continuity Salience).

Conclusion and Future Directions

This research highlights a critical pitfall in training abstractive models on highly extractive data. By introducing ILP-based copy labels, the authors provide a way to maintain abstractive quality even when the data encourages verbatim copying. Future work may involve moving from sentence-level to token-level salience identification and exploring semantic-based metrics to better capture the essence of summaries beyond simple string overlap.

Summary updated

Abstract

In text summarization, extractivity is defined as a measurement of the degree of overlap between a source document and its summary. Previous research has shown that the extractivity level of training data can influence both output extractivity and the amount of factual information (i.e. faithfulness) in outputs for abstractive summarization. However, it remains unclear if and how extractivity impacts the performance of abstractive models. In this work, we investigate the relationship between dataset extractivity and model performance by comparing the performance of trained models under different degrees of extractivity. We find that while low levels of extractivity can improve performance, as extractivity increases, performance is negatively impacted. Furthermore, through an analysis of the model’s copy continuity of content, we discover that higher extractivity leads to a greater tendency for the model to copy text continuously from the source document rather than identifying and summarizing important content that should be covered in the target summary. To address these issues, we propose a simple and effective method to design copy labels for fixing the model’s copying behaviors and train the model with a copy mechanism. The experimental results illustrate the effectiveness of our strategy in alleviating the negative impact on model performance resulting from high dataset extractivity, and that our method outperforms several competitive baselines.
BibTeX
@inproceedings{ge2023detection,
  author = {Yubin Ge and Sullam Jeoung and Ly Dinh and Jana Diesner},
  title = {{Detection and Mitigation of the Negative Impact of Dataset Extractivity on Abstractive Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.877},
}
Figure