Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers.
Introduction
In the field of natural language processing, the evaluation of text summarization has long relied on ROUGE, a metric based on n-gram overlap between a generated summary and a human-written reference. However, as modern neural models like BART and T5 produce increasingly fluent and abstractive summaries, the limitations of ROUGE have become apparent. It struggles to reward effective paraphrasing and often fails to penalize factual hallucinations. Furthermore, human evaluation protocols are fragmented, often relying on unreliable crowd-sourced workers. SummEval addresses these issues by providing a standardized, large-scale benchmark for summarization evaluation.
Methodology
The researchers employed a five-pronged approach to standardize evaluation:
- Model Collection: Outputs were gathered from 23 recent neural summarization models, including both extractive and abstractive architectures.
- Metric Benchmarking: 14 automatic metrics were evaluated, ranging from classic overlap metrics (ROUGE, BLEU) to modern embedding-based metrics (BERTScore, MoverScore).
- Human Annotation: 12,800 summary-level annotations were collected for 100 articles. Crucially, these were graded by both crowd-sourced workers and expert judges (summarization researchers).
- Evaluation Dimensions: Summaries were rated on a 1-5 scale across four axes: Coherence (logical flow), Consistency (factual accuracy), Fluency (grammaticality), and Relevance (importance of information).
- Unified Toolkit: The metrics were released as an open-source Python toolkit to facilitate standardized future research.
Key Results and Analysis
Metric Correlations with Human Judgment
A critical finding of the study is that most automatic metrics correlate poorly with human judgments of Coherence and Relevance. Interestingly, while ROUGE-L performed poorly, higher-order n-gram metrics like ROUGE-3 and classic metrics like METEOR showed surprisingly strong correlations with factual Consistency, often outperforming more complex neural metrics.
Human Evaluation Discrepancies
The study revealed a significant gap between expert and crowd-sourced evaluations. Crowd-workers exhibited a "halo effect," often giving similar scores across all four dimensions regardless of specific quality differences. In contrast, expert annotators achieved much higher inter-annotator agreement (Krippendorff’s alpha of 0.71 vs. 0.49 for crowd-workers). Furthermore, human-written reference summaries in the CNN/DailyMail dataset were often found to be of lower quality than expected, frequently containing clickbait or formatting issues that experts penalized.
Model Performance
Large-scale pretrained language models consistently outperformed other architectures. Models such as BART, Pegasus, and T5 dominated the leaderboards, particularly in terms of factual consistency and fluency.
Conclusion and Future Work
SummEval highlights the urgent need for more robust evaluation protocols in text summarization. The unreliability of crowd-sourced workers and the limitations of current automatic metrics in capturing coherence and relevance represent significant hurdles for the field. The authors recommend that future research move beyond simple n-gram overlap and develop metrics that specifically target the logical and semantic properties of summaries. They also urge the community to implement more rigorous quality controls for human evaluation to ensure progress is measured accurately.
Summary updated