To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text

Matt Wilber, William Timkey, Marten van SchijndelACL 2021

TL;DR

This paper investigates how abstractive summarization models paraphrase text, moving beyond just evaluating if they produce good summaries. By analyzing the "pointer-generator" model, they reveal that while these models can grammatically combine text, they often lack the semantic understanding for truly abstractive and faithful paraphrasing, relying more on syntactic cues and frequency biases.

What Problem Does This Paper Address?

Abstractive summarization models aim to generate summaries that go beyond simply copying text, by paraphrasing and synthesizing information. While metrics like ROUGE show improvements in generated summaries, there's a significant gap in understanding how these models achieve abstraction and what strategies they employ. Previous work has raised concerns about abstractive models producing ungrammatical or unfaithful summaries, and even repetition. This paper seeks to demystify the "abstraction switch" within these models, exploring the conditions under which they choose to generate novel text versus copy existing text, and what this reveals about their underlying capabilities and limitations.

Methods

The core of this research involves analyzing the pointer-generator model (See et al., 2017). This model is particularly interesting because it explicitly incorporates a p_gen switch, which controls the probability of generating a new word versus copying a word from the source text.

The authors employ several methods:

  • Token-level Analysis: They examine how the p_gen switch varies across a summary, correlating it with the entropy (uncertainty) of the generation and copy distributions. This helps understand when the model is "uncertain" and thus more likely to switch strategies.
  • Feature-based Modeling: They build linear models to predict p_gen using various features, including:
  • Model-internal entropies: H_gen and H_copy (uncertainty in generation/copy distributions).
  • Language model entropies: From trigram, constituency parser, and LSTM models, to measure lexical and syntactic uncertainty independent of the summarization task.
  • Structural features: Root distance and edge distance from parse trees of generated summaries, to capture syntactic boundaries.
  • Part-of-Speech (POS) tags: To see if specific word types influence the copy/generate decision.
  • Behavioral Probing: They artificially manipulate the p_gen switch to force the model to generate more abstractive text, observing the impact on summary quality and faithfulness.

Datasets

The paper uses two primary datasets:

  • CNN/DailyMail: Described as an "extractive-biased" dataset, consisting of multi-sentence summaries of news articles.
  • XSum: Described as an "abstractive-biased" dataset, where summaries are typically the first sentence of a news article. This dataset is noted as being shorter and requiring more copying from the source.

Key Insights and Ideas

  • Abstraction is often syntactic, not semantic: The pointer-generator model primarily uses its "abstraction" capability for syntactic purposes, like truncating sentences at boundaries or joining clauses, rather than for deep semantic rephrasing.
  • Uncertainty drives the switch: High uncertainty in the generation distribution (H_gen) correlates with a higher p_gen (more generation), while high certainty in the copy distribution (H_copy) also leads to higher p_gen. This suggests the model leans towards generation when it's less confident about copying or when it needs to create new text.
  • Part-of-Speech is a strong predictor: POS tags are the most significant predictors of p_gen. For instance, punctuation, commas, and conjunctions are strongly associated with generation, while numbers, proper nouns, and interjections are more likely to be copied.
  • Sentence-final punctuation is almost entirely generated: Despite being a common syntactic marker, sentence-final punctuation is almost exclusively handled by the generation component, even when it appears in novel contexts.
  • Forcing abstraction leads to unfaithfulness: Artificially increasing p_gen doesn't unlock deeper semantic understanding. Instead, it leads to more synonym substitutions and, critically, an increase in factual inaccuracies and hallucinations.
  • Dataset bias matters: The CNN/DailyMail model uses generation for stitching and punctuation, while the XSum model relies heavily on generation, only backing off to copy for proper nouns and quotes, often resulting in less grammatical or faithful summaries.

Key Results

  • CNN/DailyMail:
    • Structural features (edge and root distance) explain 20% of p_gen variance, indicating a preference for generation at larger syntactic boundaries.
  • Part-of-speech tags explain a substantial 59.3% of p_gen variance.
  • Sentence-final punctuation has a mean p_gen of 0.915, compared to the overall mean of 0.204.
  • XSum:
    • The mean p_gen is much higher (0.828) than CNN/DailyMail.
  • Linguistic features explain less variance in p_gen compared to CNN/DailyMail, suggesting a different strategy.
  • The model primarily generates novel text, with exceptions for proper nouns and quotations.
  • Forced Abstraction (CNN/DailyMail):
    • Increasing p_gen leads to more synonym substitution and a significant increase in hallucinations.
  • Even with forced generation, the model often produces simple shallow substitutions rather than structurally complex paraphrases.

Limitations and Future Work

  • Limited Semantic Understanding: The paper concludes that current abstractive models, even with explicit switches, lack the deep semantic understanding required for truly faithful and complex paraphrasing. They rely on syntactic cues and frequency biases.
  • Dataset Influence: While abstractive-biased datasets (like XSum) encourage more generation, they don't inherently grant semantic understanding, potentially leading to less faithful summaries.
  • Future Work: The authors suggest future research should focus on improving the training signal to help models better understand the underlying semantics of the source document. They also highlight the need for better evaluation metrics that can accurately assess paraphrasing ability beyond n-gram overlap. Investigating how to improve the faithfulness of abstractive summaries, especially in high-stakes domains like healthcare, remains a critical area.

Summary updated

BibTeX
@inproceedings{wilber2021point,
  author = {Matt Wilber and William Timkey and Marten van Schijndel},
  title = {{To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure