Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples

Krtin Kumar, Jackie Chi Kit CheungNAACL 2019 · OpenReview

Abstract

Krtin Kumar, Jackie Chi Kit Cheung. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

While neural abstractive summarizers are achieving record-breaking ROUGE scores, they often lack a deep semantic understanding of the texts they process. Standard evaluation metrics like ROUGE measure word overlap between machine-generated and human-written "gold" summaries. However, ROUGE is a blunt instrument; a model can achieve a high score by simply copying keywords while completely mangling the facts or the logic of the story.

This paper introduces a "stress test" using contrastive examples—slightly perturbed, incorrect versions of human summaries—to reveal that even state-of-the-art models frequently prefer factually wrong or nonsensical summaries over the original gold standards.

Methodology: The Contrastive Stress Test

The authors propose evaluating summarizers as conditional language models. Instead of just looking at what the model generates, they look at how the model scores different candidate summaries. If a model is truly effective, it should assign a higher probability to a correct human summary than to a version that has been subtly broken.

The researchers developed a rule-based procedure to generate contrastive summaries. These are "distractor" summaries created by taking a human-written gold summary and introducing a single, specific error based on Part-of-Speech (POS) tags and dependency parsing:

  1. Noun/Verb/Adjective Swaps: Replacing a word in the gold summary with another word of the same category from either the summary itself or the source text.
  2. Preposition Swaps: Switching prepositions (e.g., changing "in" to "from") when their syntactic roles match.
  3. Grammar Correction: Using SimpleNLG to ensure that when a word was swapped, the inflection was adjusted so the sentence remained grammatically plausible. This forces the model to rely on semantics rather than just spotting broken grammar.
Examples of contrastive summaries and the specific semantic failures they expose.

Evaluation Metrics

To quantify model performance, two new metrics were defined:

  • Dodged: The percentage of times the model correctly assigned a higher score to the gold summary than the contrastive one.
  • Escaped: A much tougher metric—the percentage of source texts where the model correctly "dodged" every single contrastive example generated for that text.

Key Results and Insights

The study evaluated models on two benchmark datasets: Gigaword (short headlines) and CNN/Dailymail (multi-sentence summaries). The models compared were ABS+, GTP (Pointer-Generator), and FAS (a reinforcement learning-based model).

Model Dataset Dodged (%) Escaped (%)
GTP CNN/DM 96.3% 29.8%
FAS CNN/DM 92.9% 12.2%
ABS+ Gigaword 48.6% 10.8%

The "Extractive" Crutch

Models like GTP perform well on the "Dodged" metric (over 96%) because they are "behaviorally extractive." They learn to favor words that appear in the source text. However, when the contrastive example also uses words from the source (but in the wrong order or context), the models struggle.

The Illusion of Performance

Even though models might "dodge" individual errors, their "Escaped" rates are much lower. For GTP, the escaped rate is only 29.8%, meaning that for about 70% of articles, at least one subtle word swap can trick the model into preferring an incorrect summary.

Distribution of gold summary ranks among contrastive distractors. Note the poor performance of ABS+ (Gigaword) compared to GTP and FAS.

As shown in the ranking distribution, ABS+ is surprisingly weak, essentially guessing at random (scoring ~48-50% on the Dodged metric) when faced with contrastive examples. This suggests it relies heavily on language modeling (fluency) rather than conditioning on the source text (accuracy).

Conclusion

The researchers found that models were most easily fooled by preposition and noun swaps. Human verification confirmed that 93% of the generated distractors were indeed semantically incorrect, validating the models' failures. Future work suggests using these contrastive examples during the training phase (adversarial training) to force models to pay closer attention to the relationship between the source text and the summary, rather than just learning to produce fluent-sounding English.

Summary updated

BibTeX
@inproceedings{kumar2019understanding,
  author = {Krtin Kumar and Jackie Chi Kit Cheung},
  title = {{Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=HJ-j5Qbu-S},
}
Figure