The Extractive Bias in Generative Text Summarization

85 concepts59 papers1 proposal

When we train sequence-to-sequence models for tasks like abstractive summarization or keyphrase generation, our goal is to build systems that can digest information, reason over it, and synthesize a novel response. However, a persistent pathology plagues these models: they frequently default to an extractive bias. Instead of generating original paraphrases, they act as sophisticated copy-paste engines, lifting long, verbatim sequences directly from the source text.

This behavior is not just a stylistic quirk; it represents a fundamental optimization failure. To understand why models default to copying, we have to look at the intersection of training dynamics, dataset design, and evaluation metrics.

The Illusion of Abstraction

Early neural summarization architectures struggled to move past simple "cut-and-paste" extraction, as noted by Sumit Chopra et al. (2016). Even with the advent of massive Transformers and large language models, this tendency remains. In fact, Fangfang Zhang et al. (2018) demonstrated that a "pure copy" baseline—which is constrained to only select words from the source text—can perform competitively with complex abstractive models while being significantly faster.

This happens because modern generative models often reuse a high percentage of full sentences or long n-grams from the source document, creating a substantial quantitative gap between machine-generated text and the highly novel vocabulary found in human writing (Pengcheng Liao et al. (2019)). This bias extends beyond summarization into related generative tasks; for instance, neural keyphrase generation models frequently fail to produce "unseen" vocabulary, limiting their utility in expanding document indexes for information retrieval (Florian Boudin et al. (2021)).

Why Models Take the Extractive Shortcut

Why does a highly parameterized neural network, theoretically capable of generating any sequence of tokens, choose to copy? The answer lies in the path of least resistance during optimization.

1. The Early-Training Loss Trap

During the initial phases of fine-tuning, a model faces a massive search space of potential words. Tanya Goyal et al. (2022) uncovered a critical phenomenon in training dynamics: models learn to copy input text very early because it offers an immediate, massive reduction in training loss.

Learning a pointer-like alignment or copying mechanism is mathematically simple. Once the model discovers this shortcut, it hits an "extractive overfitting" plateau. The immediate loss reduction from copying prevents the model from exploring the more complex, sparse gradient paths required to develop abstractive reasoning and creative paraphrasing (Xiaoyu Shen et al. (2019)).

2. Dataset Echo Chambers

Neural networks are statistical mirrors of their training data. Many standard evaluation benchmarks and reference datasets suffer from high inherent extractivity (Moussa Kamal Eddine et al. (2021)). When target summaries are largely composed of verbatim snippets from the original document, models are actively penalized for abstractive divergence (Xiaojun Liu et al. (2022)). Even when datasets are derived from expert human annotations, they often retain this extractive bias, training models to prioritize sentence rearrangement over genuine synthesis (Daniela Brook Weiss et al. (2022)).

3. The Word-Overlap Reward Problem

Standard training objectives like Maximum Likelihood Estimation (MLE) evaluate tokens based on exact matches. This alignment with token-level overlap metrics (such as ROUGE) actively discourages lexical novelty. If a model attempts a creative paraphrase using a synonym not present in the reference text, it receives a lower reward than if it simply copied the source text verbatim (Romain Paulus (2018)). Similarly, pre-training objectives that rely heavily on word-overlap metrics to identify salient sentences condition the model to equate syntactic similarity with semantic importance (Alireza Salemi et al. (2021)).

Where the Shortcut Fails

The extractive shortcut works well on standard news datasets where the first few sentences contain the core message. However, this bias causes catastrophic failures when tasks require deeper cognitive processing.

  • Creative and Figurative Language: When summarizing poetry, models struggle with metaphorical depth and default to literal copy-pasting of lines, failing to provide a true thematic overview (Ridwan Mahbub et al. (2023)).
  • Inferential Reasoning: When asked to explain character motivations in a narrative, models copy existing text segments rather than synthesizing an explanation based on underlying goals and plans (Yash Kumar Lal et al. (2021)).
  • Evidence Synthesis: In multi-document biomedical summarization, large language models frequently default to paraphrasing introductory background prompts rather than aggregating and synthesizing the actual experimental results from the source documents (Yulia Otmakhova et al. (2022)).
  • Zero-Shot Settings: Without explicit fine-tuning or massive scale, generative models in zero-shot settings heavily default to extractive behavior, often failing to outperform simple baselines like selecting the lead sentence of a document (Antoine Simoulin et al. (2021)).

This copying behavior also creates an evaluation blindspot. Because models reuse source text verbatim, simple lexical overlap measures remain artificially competitive with sophisticated neural verification methods, masking the need for metrics that can handle true semantic paraphrasing (Daniel Deutsch et al. (2022)).

Breaking the Copy-Paste Habit

To force models off the extractive plateau, researchers employ strategies that disrupt the easy copying shortcut at both the data and algorithmic levels.

Data-Level Interventions

One straightforward approach is to eliminate the statistical cues that models exploit. By purging samples where the summary is a verbatim copy of the lead sentences, researchers prevent models from adopting simple positional heuristics (Lokesh Madasu (2023); Ashok Urlana et al. (2023)). Additionally, data augmentation techniques can inject lexical diversity; for example, the authors of ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization combine rule-based extraction with round-trip translation to generate high-quality, paraphrased pseudo-training data that breaks the syntactic alignment between source and target.

Algorithmic and Objective Constraints

Rather than changing the data, other methods modify the model's optimization landscape to penalize copying:

Standard Training:
[Source Text] ---> (Model minimizes token loss) ---> [Learns easy copying shortcut]

Loss Truncation / Constraints:
[Source Text] ---> (Model copies easy tokens) ---> [Loss Truncation drops "easy" tokens]
                                              ---> [Model forced to learn abstractive synthesis]
  • Loss Truncation: To combat the early-training trap, Tanya Goyal et al. (2022) proposed dropping tokens with very low loss during training. Because copied tokens are "easy" and have low loss, truncating them forces the network to focus its gradient updates on the harder, abstractive tokens.
  • Novelty Rewards: In reinforcement learning frameworks, models can be explicitly penalized for copying long sequences from the source text via a novelty reward, forcing the decoder to find alternative phrasings (Romain Paulus (2018)).
  • Bottom-Up Attention Masking: Sebastian Gehrmann (2018) introduced a content selector that restricts the decoder's attention to only a sparse set of important source phrases. By blocking the decoder from seeing the full, continuous source text, the model is structurally prevented from lifting long, verbatim chunks, forcing it to compress and synthesize information.

Key Foundational Work

Abstractive Sentence Summarization with Attentive Recurrent Neural Networks (2016) — contributed 1 concept elements to this direction.
Decoupling Encoder and Decoder Networks for Abstractive Document Summarization (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_extractive_bias_454eca02,
  title = {extractive bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extractive-bias/rdr_454eca02afa94c3ba96f364ad7378033},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_extractive_bias_454eca02_v1,
  title = {extractive bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extractive-bias/rdr_454eca02afa94c3ba96f364ad7378033/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake