Abstract
An abstract must not change the meaning of the original text. A single most effective way to achieve that is to increase the amount of copying while still allowing for text abstraction. Human editors can usually exercise control over copying, resulting in summaries that are more extractive than abstractive, or vice versa. However, it remains poorly understood whether modern neural abstractive summarizers can provide the same flexibility, i.e., learning from single reference summaries to generate multiple summary hypotheses with varying degrees of copying. In this paper, we present a neural summarization model that, by learning from single human abstracts, can produce a broad spectrum of summaries ranging from purely extractive to highly generative ones. We frame the task of summarization as language modeling and exploit alternative mechanisms to generate summary hypotheses. Our method allows for control over copying during both training and decoding stages of a neural summarization model. Through extensive experiments we illustrate the significance of our proposed method on controlling the amount of verbatim copying and achieve competitive results over strong baselines. Our analysis further reveals interesting and unobvious facts.
Introduction
In the field of AI-driven text summarization, a constant tension exists between extractive methods (copying key sentences verbatim) and abstractive methods (rephrasing ideas). While abstractive models are more human-like, they often suffer from "hallucinations"—generating facts not present in the source. Conversely, purely extractive models can be rigid and may lead to copyright concerns if they copy large blocks of text.
This research introduces CopyTrans, a decoder-only Transformer model that provides a "dial" for copying behavior. By training on single human abstracts, the model learns to produce a wide spectrum of summaries, from purely extractive to highly generative, allowing users to fine-tune the output based on their specific needs for accuracy or creativity.
Methods: The CopyTrans Architecture
CopyTrans moves away from traditional encoder-decoder architectures, framing summarization as a language modeling task using a decoder-only Transformer. This unified setup allows the model to use the same Transformer blocks to both "read" the source text and "write" the summary, facilitating more effective warm-starting with pre-trained weights like BERT.
Strategic Masking and Training
The core innovation is a strategic masking process during training. The model is trained to predict three types of tokens:
- Seen tokens: Words appearing in both the source and the summary.
- Unseen tokens: Words unique to the summary.
- Source tokens: Words from the original source text.
By varying the proportions of these tokens during training, the model can be steered toward different levels of abstraction. To further refine the output at runtime, the researchers introduced Soft-bounded word reward (SBWR), a reranking step that balances summary length with information density.
Results and Analysis
The model was evaluated on the Gigaword and Newsroom benchmarks, demonstrating superior performance and flexibility compared to strong baselines like Pointer-Generator networks and BiSET.
Performance Comparison
On the Gigaword test set, CopyTrans achieved state-of-the-art results. The variant utilizing Beam Search with SBWR achieved a ROUGE-2 score of 20.47, significantly surpassing previous models like BiSET (19.53) and PG Networks (16.92).
Key Insights
- The "Seen vs. Unseen" Driver: The ratio of seen to unseen words during training is the primary factor in determining the model's copying behavior.
- Source Training Benefits: Predicting missing words in the source text improves the model's representation learning, leading to higher overall ROUGE scores.
- Human Evaluation: In "Best-Worst Scaling" tests, human evaluators preferred CopyTrans summaries over human-written abstracts for their informativeness, grammaticality, and truthfulness.
Conclusion
CopyTrans provides a robust framework for controlling the verbatim copying rate in abstractive summarization. By leveraging a decoder-only architecture and strategic masking, the model achieves competitive results while offering the flexibility to generate summaries across the extractive-abstractive spectrum. Future work will explore scaling this approach to multi-paragraph documents and implementing more sophisticated search heuristics.
Summary updated