SummEval: Re-evaluating Summarization Evaluation

Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir R. Radev

Abstract

The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers.

Introduction

In the field of natural language processing, the evaluation of text summarization has long relied on ROUGE, a metric based on n-gram overlap between a generated summary and a human-written reference. However, as modern neural models like BART and T5 produce increasingly fluent and abstractive summaries, the limitations of ROUGE have become apparent. It struggles to reward effective paraphrasing and often fails to penalize factual hallucinations. Furthermore, human evaluation protocols are fragmented, often relying on unreliable crowd-sourced workers. SummEval addresses these issues by providing a standardized, large-scale benchmark for summarization evaluation.

Methodology

The researchers employed a five-pronged approach to standardize evaluation:

  1. Model Collection: Outputs were gathered from 23 recent neural summarization models, including both extractive and abstractive architectures.
  2. Metric Benchmarking: 14 automatic metrics were evaluated, ranging from classic overlap metrics (ROUGE, BLEU) to modern embedding-based metrics (BERTScore, MoverScore).
  3. Human Annotation: 12,800 summary-level annotations were collected for 100 articles. Crucially, these were graded by both crowd-sourced workers and expert judges (summarization researchers).
  4. Evaluation Dimensions: Summaries were rated on a 1-5 scale across four axes: Coherence (logical flow), Consistency (factual accuracy), Fluency (grammaticality), and Relevance (importance of information).
  5. Unified Toolkit: The metrics were released as an open-source Python toolkit to facilitate standardized future research.

Key Results and Analysis

Metric Correlations with Human Judgment

A critical finding of the study is that most automatic metrics correlate poorly with human judgments of Coherence and Relevance. Interestingly, while ROUGE-L performed poorly, higher-order n-gram metrics like ROUGE-3 and classic metrics like METEOR showed surprisingly strong correlations with factual Consistency, often outperforming more complex neural metrics.

Table 4: Spearman correlation of automatic metrics with expert human judgments across four dimensions.

Human Evaluation Discrepancies

The study revealed a significant gap between expert and crowd-sourced evaluations. Crowd-workers exhibited a "halo effect," often giving similar scores across all four dimensions regardless of specific quality differences. In contrast, expert annotators achieved much higher inter-annotator agreement (Krippendorff’s alpha of 0.71 vs. 0.49 for crowd-workers). Furthermore, human-written reference summaries in the CNN/DailyMail dataset were often found to be of lower quality than expected, frequently containing clickbait or formatting issues that experts penalized.

Model Performance

Large-scale pretrained language models consistently outperformed other architectures. Models such as BART, Pegasus, and T5 dominated the leaderboards, particularly in terms of factual consistency and fluency.

Table 5: Human evaluation results for various extractive and abstractive models across the four quality dimensions.

Conclusion and Future Work

SummEval highlights the urgent need for more robust evaluation protocols in text summarization. The unreliability of crowd-sourced workers and the limitations of current automatic metrics in capturing coherence and relevance represent significant hurdles for the field. The authors recommend that future research move beyond simple n-gram overlap and develop metrics that specifically target the logical and semantic properties of summaries. They also urge the community to implement more rigorous quality controls for human evaluation to ensure progress is measured accurately.

Summary updated

Abstract

The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
BibTeX
@article{fabbri2021summeval,
  author = {Alexander R. Fabbri and Wojciech Kryscinski and Bryan McCann and Caiming Xiong and Richard Socher and Dragomir R. Radev},
  title = {{SummEval: Re-evaluating Summarization Evaluation}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=lESzxdAJ4oY},
}
Figure