TL;DR
This paper reveals that standard ROUGE F1 scores are misleading for evaluating neural summarization models because they don't account for summary length, which has a non-linear impact on scores. The authors propose a novel normalization method using a random baseline to provide a more accurate comparison of summarization quality, independent of length.
What Problem Does This Paper Address?
Traditionally, summarization systems were evaluated by comparing summaries of a fixed target length. However, the advent of neural summarization models has led to systems that produce summaries of variable lengths. This shift has created a problem: standard evaluation metrics like ROUGE F1 scores are sensitive to summary length. The paper highlights that as summary length increases, ROUGE F1 scores initially rise steeply and then gradually decline. This non-linear relationship means that a system producing a longer summary might appear better simply due to its length, rather than its actual summarization quality, confounding the evaluation of new techniques. The authors aim to address this by proposing a method to normalize ROUGE scores, making comparisons between systems of different lengths more equitable and insightful.
Methods
The core of the paper's contribution lies in its proposed evaluation method. Instead of directly using ROUGE F1 scores, the authors suggest normalizing them. Their approach involves dividing a system's ROUGE F1 score by the ROUGE F1 score of a random system that produces summaries of the same average output length. This normalization serves to isolate the quality of content selection from the effect of summary length. The random baseline is chosen because its output length is controllable, and any system is expected to perform at least as well as a random one in terms of content selection. They also explore representing summaries in semantic space using various sentence encoders and word embeddings to see if length still confounds similarity-based evaluations.
Datasets
The primary dataset used for experiments and evaluation is the CNN/DailyMail news dataset. This dataset is a standard benchmark for summarization tasks. The authors also refer to the Newsroom dataset in their human evaluation section.
Key Insights and Ideas
- Length is a confounding factor in ROUGE F1: The paper convincingly demonstrates that ROUGE F1 scores are not a pure measure of summarization quality but are heavily influenced by summary length, exhibiting a non-linear relationship.
- Neural models can "game" ROUGE: The variable length output of neural models can lead to misleading conclusions, potentially favoring systems that learn to produce longer summaries rather than better ones.
- Normalization is crucial: A normalized ROUGE score, using a random baseline, provides a more accurate and fair comparison of summarization systems, disentangling content selection from length.
- Semantic similarity is also length-dependent: Even when using sophisticated embedding methods, summary-reference similarity can be influenced by length, suggesting the problem is not unique to word overlap metrics.
- Human evaluations also need normalization: Pilot human evaluations show similar trends, with longer summaries often preferred for informativeness, indicating that human judgments also require normalization for fair comparison.
Key Results
- ROUGE F1 curves are non-linear: For all tested systems (lead, random, TextRank, Pointer-Generator), ROUGE F1 scores initially increase with summary length and then gradually decrease.
- Random systems perform well with length: Even a random system can achieve better ROUGE scores if allowed to produce longer summaries, highlighting the impact of length.
- Normalization reorders systems: Re-analysis of 16 systems on the CNN/DailyMail dataset using the proposed normalized ROUGE scores leads to significant changes in system rankings compared to standard ROUGE F1 scores. For instance, the
latent_cmprsystem, which was ranked lower by plain ROUGE, emerges as the third most effective after normalization. - Human ratings also show length bias: Human evaluations for informativeness correlate positively with summary length, suggesting that normalization is also needed for human judgments to be meaningful.
Limitations and Future Work
- Pilot human evaluation: The human evaluation section is described as a "pilot analysis," suggesting it's a small-scale study and more extensive human evaluations are needed.
- Need for content precision evaluation: The authors emphasize the need for human evaluations that specifically assess content precision, which is crucial for fully evaluating abstractive summarization systems.
- Future work: The paper suggests that future human evaluations should focus on judgments that capture precision in content selection, as current methods might be missing this aspect. They also imply that their normalization method can be applied to other evaluation metrics and human judgment dimensions.
Summary updated