SummEval: Re-evaluating Summarization Evaluation

Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir R. Radev

TL;DR

Text summarization evaluation is currently a mess: everyone relies on the standard ROUGE metric, but nobody agrees on how to properly measure true summary quality. This paper introduces SummEval, a massive benchmark that re-evaluates 14 automatic metrics against outputs from 23 modern neural models and expert human annotations. The authors reveal that crowd-sourced human evaluations are highly unreliable, the "ground truth" datasets are flawed, and the NLP community desperately needs new metrics to measure a summary's coherence and relevance.

What Problem Does This Paper Address?

If you build an AI that summarizes news articles, how do you prove yours is the best? For years, the default answer has been ROUGE, a metric that simply counts how many words or phrases overlap between the AI's summary and a human-written reference.

However, modern neural networks (like BART or T5) generate highly fluent, paraphrased text. ROUGE struggles to reward good paraphrasing and often fails to penalize factual hallucinations. While newer, AI-driven metrics (like BERTScore or MoverScore) have been proposed, researchers rarely compare them consistently. Furthermore, when researchers do use human judges to evaluate their models, they usually rely on cheap crowd-sourcing (like Amazon Mechanical Turk). The authors of this paper noticed these fragmented, inconsistent evaluation protocols were actively harming progress in the field, prompting them to conduct a massive, standardized re-evaluation of how we evaluate summarizers.

Methods

To clean up the evaluation landscape, the authors took a comprehensive, five-pronged approach:

  1. Model Collection: They gathered actual summary outputs from 23 recent neural summarization models (both extractive models that copy-paste sentences, and abstractive models that generate new text).
  2. Metric Benchmarking: They evaluated all these outputs using 14 different automatic metrics, ranging from classic n-gram overlap (ROUGE, BLEU, METEOR) to modern embedding-based metrics (BERTScore, MoverScore).
  3. Human Annotation: They collected 12,800 summary-level human annotations on 100 articles. Crucially, they had the summaries graded by both crowd-sourced workers and expert judges (researchers who have published in summarization).
  4. Four Dimensions of Quality: Humans graded the summaries on a 1-5 scale across four axes: Coherence (how well it flows), Consistency (factual accuracy), Fluency (grammar), and Relevance (did it pick the most important info?).
  5. Unified Toolkit: They packaged all 14 metrics into a single, easy-to-use open-source Python toolkit to standardize future research.

Datasets

  • CNN/DailyMail: The standard, widely-used dataset for training and testing news summarization models.
  • SummEval (New Contribution): The authors released their massive collection of aligned model outputs and the accompanying 12,800 expert and crowd-sourced human annotations. This serves as a new benchmark for testing future evaluation metrics.

Key Insights and Ideas

  • Crowd-workers are unreliable judges: The study found zero correlation between expert scores and crowd-sourced scores. Crowd-workers tended to give a summary the same score across all four categories (a "halo effect"), failing to distinguish between a summary that is fluent but factually wrong, versus one that is factual but poorly written.
  • The "Ground Truth" is flawed: Surprisingly, the human-written reference summaries in the CNN/DailyMail dataset scored poorly when evaluated by experts. They often contained clickbait, hyperlinks, or were just bullet points mashed together, which experts penalized for poor coherence and relevance.
  • Metrics fail at Coherence and Relevance: Almost all automatic metrics struggled to correlate with human judgments on Coherence and Relevance. Current metrics are mostly based on word/token alignment, which doesn't capture the logical flow between consecutive sentences.
  • ROUGE isn't entirely dead: While ROUGE-L (the most commonly reported variant) performed poorly, higher-order ROUGE metrics (like ROUGE-3, which looks at 3-word chunks) actually correlated surprisingly well with factual consistency.

Key Results

  • Inter-annotator agreement: Expert annotators achieved a solid agreement score (Krippendorff’s alpha of 0.71) after two rounds of review. Crowd-sourced workers had a poor agreement score of 0.49.
  • Model Rankings: Massive pretrained language models (BART, Pegasus, T5) consistently dominated the human evaluation leaderboards, particularly scoring the highest in factual consistency and fluency.
  • Metric Correlations: When compared against expert human judgments, METEOR and ROUGE-3 showed the strongest correlations for Factual Consistency (0.6324 and 0.7059, respectively), actually beating out newer, complex neural metrics in this specific dimension.

Limitations and Future Work

The authors acknowledge that their study is limited to the CNN/DailyMail dataset, which consists entirely of news articles. The performance of these metrics and models might look very different in other domains, such as summarizing scientific papers, legal documents, or dialogue.

For future work, the authors strongly urge the community to stop relying solely on crowd-workers for human evaluation without rigorous quality control. They also highlight a massive open opportunity for researchers to develop new automatic metrics that specifically target document coherence and relevance, as these remain unsolved challenges.

Summary updated

Abstract

The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently bench-mark 23 recent summarization models us-ing the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
BibTeX
@article{fabbri2021summeval,
  author = {Alexander R. Fabbri and Wojciech Kryscinski and Bryan McCann and Caiming Xiong and Richard Socher and Dragomir R. Radev},
  title = {{SummEval: Re-evaluating Summarization Evaluation}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=lESzxdAJ4oY},
}

Loading related content...

Figure