Neural Text Summarization: A Critical Evaluation

Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher

TL;DR

This paper critically evaluates the current state of neural text summarization, arguing that progress has stagnated due to fundamental issues with datasets, evaluation metrics, and model biases. The authors propose that the field needs a more robust research setup to truly advance beyond current limitations.

What Problem Does This Paper Address?

Neural text summarization aims to automatically generate concise, human-readable summaries from longer documents. While there's been a surge of interest and numerous models developed, the authors observe that performance on standard benchmarks has plateaued. They contend that this stagnation isn't due to a lack of model innovation, but rather inherent flaws in the research ecosystem itself. Specifically, they point to:

  1. Underconstrained Datasets: Datasets are often automatically collected, leading to noise and ambiguity that make the summarization task ill-defined for end-to-end models.
  2. Weak Evaluation Protocols: Current metrics, primarily ROUGE, have a weak correlation with human judgment and fail to capture crucial aspects like factual correctness.
  3. Model Biases: Models tend to overfit to specific characteristics of datasets, such as the "inverted pyramid" structure common in news articles, limiting their generalizability and output diversity.

Methods

The paper doesn't introduce new models or algorithms. Instead, it employs a critical analysis and empirical investigation of existing components of the summarization research setup. Their methodology involves:

  • Human Studies: Conducting experiments where human annotators are asked to identify important sentences in articles, both with and without specific guiding questions (constrained vs. unconstrained summarization). This helps quantify the ambiguity in content selection and the impact of task framing.
  • Dataset Analysis: Examining existing datasets (CNN/DM, Newsroom) for noise and identifying common patterns of flawed reference summaries.
  • Metric Correlation Analysis: Evaluating how well ROUGE scores correlate with human judgments across various dimensions (relevance, consistency, fluency, coherence) using data from the CNN/DM dataset.
  • Model Output Analysis: Comparing the outputs of various state-of-the-art summarization models using ROUGE scores, both against original reference summaries and against a "Lead-3" baseline (which simply extracts the first three sentences). This helps reveal model biases and the extent to which they exploit dataset artifacts.

Datasets

The paper primarily discusses and analyzes existing datasets:

  • CNN/DailyMail (CNN/DM): A widely used news corpus for both abstractive and extractive summarization.
  • Newsroom: Another large news corpus.
  • Gigaword: Used for headline generation and adapted for summarization.
  • NYT: Articles from the New York Times magazine with librarian-written abstracts.
  • XSum: Articles with single-sentence summaries, targeting abstractive models.
  • Reddit TIFU: Posts from Reddit with user-generated TL;DR summaries.
  • WikiHow: Articles from WikiHow with procedural instructions.

The authors highlight issues with the automatic collection and inherent noise within these datasets, particularly CNN/DM and Newsroom.

Key Insights and Ideas

  • The "Underconstrained Task": The current setup, where models are given an article and a single reference summary without further guidance, makes the task inherently ambiguous. Human annotators show significantly better agreement on important sentences when provided with specific questions to answer in their summaries.
  • Layout Bias is Dominant: Models heavily exploit the "inverted pyramid" structure of news articles, where the most important information is at the beginning. This bias is so strong that using the first three sentences (Lead-3 baseline) as a reference significantly boosts ROUGE scores for most models, suggesting they are not truly learning to summarize content but rather to mimic this layout.
  • ROUGE is a Flawed Metric: ROUGE scores show a weak correlation with human judgments, especially for abstractive models. It fails to capture factual consistency, a critical aspect of summary quality, and its correlation can even decrease with more reference summaries.
  • Noise in Datasets is Pervasive: A non-trivial percentage of reference summaries in common datasets contain errors, irrelevant links, or placeholder text, which can negatively impact model training and evaluation.
  • Limited Model Diversity: The analysis of model outputs reveals that many models produce very similar summaries, suggesting they are all overfitting to similar patterns in the training data rather than developing diverse summarization strategies.

Key Results

  • Human annotators agreed on the importance of significantly more sentences per article in a constrained setting (where they answered specific questions) compared to an unconstrained setting. For example, with a threshold of 3 human votes, annotators agreed on 1.39 sentences in the constrained setting versus 0.63 in the unconstrained setting.
  • When ROUGE scores were computed against the Lead-3 baseline instead of the original reference summaries, most models showed a substantial increase in scores, highlighting their reliance on the article's beginning.
  • A manual inspection of 200 abstractive model summaries revealed that 30% contained factual inconsistencies.
  • Pearson correlation coefficients between ROUGE scores and human judgments for abstractive models were generally very low (e.g., 0.04 for relevance, 0.01 for consistency), with slightly better but still weak correlations for extractive models.

Limitations and Future Work

Limitations:

  • The study acknowledges that the human studies, while aiming for high quality, involve a trade-off with the number of annotated examples.
  • The analysis of ROUGE correlation was conducted on a specific dataset (CNN/DM) and may not generalize perfectly to all domains.

Future Work:

The authors implicitly suggest several directions:

  • Developing more constrained datasets: Datasets that provide clearer guidance on what information is important would lead to better-defined tasks.
  • Improving evaluation metrics: The field needs metrics that go beyond n-gram overlap and can assess factual correctness, fluency, and relevance more robustly.
  • Designing models less susceptible to layout bias: Future models should be trained to generalize across different document structures and not just exploit news article conventions.
  • Encouraging diversity in model outputs: Research should focus on generating more varied and creative summaries, rather than converging on similar outputs.

Summary updated

Abstract

Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original document. Despite increased interest in the community and notable research effort, progress on benchmark datasets has stagnated. We critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary shortcomings: 1) automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited diversity in their outputs.
BibTeX
@inproceedings{kryscinski2019neural,
  author = {Wojciech Kryscinski and Nitish Shirish Keskar and Bryan McCann and Caiming Xiong and Richard Socher},
  title = {{Neural Text Summarization: A Critical Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=0EBqmkxp5W},
}
Figure