Abstract
Maxime Peyrard. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
2019.
Introduction
The field of Natural Language Processing (NLP) relies heavily on automatic metrics like ROUGE to benchmark progress in text summarization. These metrics are traditionally validated by measuring their correlation with human judgments. However, a significant distribution shift has occurred: most human-annotated datasets originate from the DUC/TAC shared tasks of 2008–2009, an era when summarization systems were considerably weaker than today's models.
Modern systems frequently outperform the best legacy systems, yet we continue to evaluate them using metrics validated on a lower performance tier. This research investigates whether these metrics remain reliable in the "high-scoring range" where contemporary research operates.
Methodology
To analyze metric behavior in the high-scoring range (T), where human annotations are scarce, the study employed a "stress test" approach:
- Optimization: A genetic algorithm was used to evolve summaries that maximize specific metrics (e.g., ROUGE-2, ROUGE-L, JS-2). This generated a diverse population of summaries, from low-quality to "theoretically perfect" according to specific metrics.
- Filtering: Summaries scoring higher than a LexRank baseline were categorized as the "high-scoring range."
- Pairwise Correlation: The study compared metrics against each other. The premise is that if multiple metrics are valid, they should maintain a high degree of consensus even as summary quality improves.
The analysis utilized classic benchmarks (TAC-2008/2009) for the "average-scoring range" (A) and a synthetic dataset of 160,523 summaries for the high-scoring range (T).
Results and Discussion
The most significant finding is a phenomenon akin to Simpson’s Paradox: while metrics agree on the relative quality of summaries across the entire performance spectrum, this consensus vanishes when focusing solely on high-quality summaries.
Correlation Collapse
In the average range (A), metrics like ROUGE-2 and JS-2 exhibit strong correlation (Kendall’s $\tau \approx 0.88$). However, in the high-scoring range (T), this correlation plummets to 0.28. This suggests that as systems improve, our standard tools lose their ability to distinguish between them reliably.
Metric Divergence and Negative Trends
The study found that ROUGE-L (Longest Common Subsequence) exhibits a negative correlation with other metrics in the high-scoring range. Specifically, its correlation with ROUGE-2 drops from 0.53 in the average range to -0.18 in the high range. This implies that optimizing for ROUGE-L might inadvertently degrade summary quality according to other established measures.
The Fallacy of Multiple Metrics
Reporting ROUGE-1, ROUGE-2, and ROUGE-L together is standard practice, intended to provide a robust evaluation. However, the data shows that in the high-scoring range, the overlap—summaries that all metrics agree are improvements—drops to nearly zero. Consequently, reporting multiple metrics may simply be reporting different types of noise rather than a consensus of quality.
Conclusion
The study highlights a "metric crisis" at the top of the performance scale. As summary quality increases, the percentage of metric disagreement climbs steadily. While the use of genetic algorithms for summary generation provides a useful upper bound, the linguistic properties may differ from neural architectures.
The findings serve as a "call to arms" for the NLP community: there is an urgent need for new human-annotated datasets targeting the high-scoring range. Without such benchmarks, research risks optimizing for metrics that no longer align with human perception at the frontier of the field.
Summary updated