Controlling the Amount of Verbatim Copying in Abstractive Summarization

Kaiqiang Song, Bingqing Wang, Zhe Feng, Ren Liu, Fei Liu

Introduction

Modern AI-driven text summarization faces a persistent challenge: balancing extraction (verbatim copying) with abstraction (generating novel phrasing). While abstractive models aim for human-like fluency, they are prone to "hallucinations"—generating factual inaccuracies. Conversely, excessive copying can lead to clunky summaries or legal issues, such as copyright infringement under regulations that restrict long verbatim excerpts.

This research introduces CopyTrans, a framework designed to "turn the dial" on copying behavior. By providing a flexible mechanism to adjust the degree of verbatim extraction, the authors enable the generation of summaries ranging from purely extractive to highly generative, tailored to specific needs.

Methodology: The CopyTrans Framework

The authors depart from the traditional encoder-decoder setup, opting instead for a decoder-only Transformer architecture. This unified approach treats summarization as a language modeling task where both the source text and the summary are processed within the same network.

Unified Architecture and Pre-training

By using a single Transformer, the model can leverage pre-trained representations (such as BERT) to "warm-start" the entire network. This includes the attention mechanisms, a feat typically difficult to achieve in standard encoder-decoder configurations.

"Mix-and-Match" Training

A core innovation is the training strategy. The model is trained to predict masked tokens categorized into three groups: (a) summary words seen in the source, (b) summary words unseen in the source, and (c) source words. By varying the proportions of these categories during training, the researchers can precisely control the model's tendency to copy or generate.

The CopyTrans architecture uses a masked input strategy to train the model on seen and unseen words.

Controlled Decoding

To further refine summaries during inference, the authors introduced two reranking strategies: BP-norm (Brevity Penalty) and SBWR (Soft-bounded word reward). These allow for fine-tuning summary length and copy rates on the fly without retraining.

Experimental Results

The model was evaluated on two major benchmarks: Gigaword (4 million news headlines) and Newsroom (1.3 million diverse articles).

State-of-the-Art Performance

CopyTrans achieved significant improvements over existing baselines. On the Gigaword dataset, the model reached a ROUGE-L score of 36.69, outperforming established models like PG-Networks and BiSET.

Comparison of CopyTrans variants against baseline systems on the Gigaword dataset.

Key Insights

  • The "Seen vs. Unseen" Lever: The ratio of extractive ("seen") to abstractive ("unseen") words during training is the primary driver of copying behavior.
  • Representation Learning: Predicting source words does not alter copying behavior but enhances the model's internal understanding, leading to higher ROUGE scores.
  • Robustness: Models with higher copy rates proved more robust when transferring between different domains (e.g., from Gigaword to Newsroom), whereas highly abstractive models were more sensitive to vocabulary shifts.

Human Evaluation

In human assessments for informativeness, grammaticality, and truthfulness, the "best-abstract" version of CopyTrans consistently outperformed popular Pointer-Generator (PG) networks, demonstrating that controlled abstraction does not come at the cost of quality.

Conclusion and Future Work

CopyTrans provides a powerful tool for navigating the trade-off between extraction and abstraction. While human-written references remain an imperfect "ground truth" due to their own occasional inaccuracies, this framework offers a path toward more reliable and legally compliant summarization. Future work may explore more advanced decoding algorithms, such as A* search, to further optimize the balance between summary quality and specific copy constraints.

Summary updated

Abstract

An abstract must not change the meaning of the original text. A single most effective way to achieve that is to increase the amount of copying while still allowing for text abstraction. Human editors can usually exercise control over copying, resulting in summaries that are more extractive than abstractive, or vice versa. However, it remains poorly understood whether modern neural abstractive summarizers can provide the same flexibility, i.e., learning from single reference summaries to generate multiple summary hypotheses with varying degrees of copying. In this paper, we present a neural summarization model that, by learning from single human abstracts, can produce a broad spectrum of summaries ranging from purely extractive to highly generative ones. We frame the task of summarization as language modeling and exploit alternative mechanisms to generate summary hypotheses. Our method allows for control over copying during both training and decoding stages of a neural summarization model. Through extensive experiments we illustrate the significance of our proposed method on controlling the amount of verbatim copying and achieve competitive results over strong baselines. Our analysis further reveals interesting and unobvious facts.
BibTeX
@inproceedings{song2020controlling,
  author = {Kaiqiang Song and Bingqing Wang and Zhe Feng and Ren Liu and Fei Liu},
  title = {{Controlling the Amount of Verbatim Copying in Abstractive Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6420},
}
Figure