TL;DR
This paper dives deep into popular text summarization datasets, revealing that they are far from perfect. The authors introduce a novel typology to categorize data quality and complexity, finding that current automatic evaluation metrics struggle to reliably assess summaries from noisy or complex datasets, potentially leading to misleading performance evaluations.
What Problem Does This Paper Address?
State-of-the-art text summarization systems are trained and evaluated on massive datasets scraped from the web. While these datasets have enabled significant progress, there's a critical lack of understanding about their underlying characteristics. Specifically, researchers don't fully grasp the extent of "noise" (e.g., missing facts, irrelevant information) or the varying levels of summarization complexity (e.g., simple extraction vs. abstractive paraphrasing) present in these datasets. This knowledge gap hinders our ability to interpret model performance, understand the reliability of automatic evaluation metrics like ROUGE, and ultimately, to build better summarization systems.
Methods
The core of this research involves a meticulous manual analysis of 600 samples from three prominent summarization datasets: Gigaword, CNN/DailyMail (CNN/DM), and XSum. The authors developed a six-class typology to categorize samples based on two main dimensions:
- Data Noise: Incomplete/Irrelevant, Entity Missing, Evidence Missing.
- Summarization Complexity: Extractive, Paraphrase, Inference.
Two authors independently annotated the samples, with disagreements resolved through discussion. This annotated data was then used to evaluate the performance of 27 state-of-the-art summarization models and 5 popular automatic metrics across these different classes. The study also investigated the correlation between different metrics and, for CNN/DM, correlation with human judgments.
Datasets
The paper focuses on three widely used summarization datasets:
- Gigaword: Primarily used for headline generation, characterized by short sources and references.
- CNN/DailyMail (CNN/DM): A popular dataset of news articles paired with human-generated summaries.
- XSum: Designed for highly abstractive summarization, with very short summaries.
No new datasets were introduced; instead, the research provides a novel analytical framework for existing ones.
Key Insights and Ideas
- Datasets are Not Uniform: Each dataset has a distinct distribution of data quality and complexity, largely influenced by their collection processes. Gigaword is noisy and extractive, XSum is abstractive but very noisy, and CNN/DM is relatively cleaner and more abstractive.
- Noise is a Major Issue: A significant portion of samples across datasets suffer from noise (missing facts, irrelevant content), which can unfairly penalize models during evaluation. Gigaword has 44.5% noise, while XSum has 55%.
- Complexity Matters for Metrics: The reliability of automatic metrics is heavily dependent on the complexity of the samples. Metrics show higher agreement with human judgments on simpler (extractive) samples compared to more complex (paraphrase, inference) ones.
- "Summarization-ese" Bias: Token-based metrics like ROUGE might be biased towards simpler, more "extractive" summaries, potentially overlooking the nuances of abstractive generation.
- Human Judgment is Crucial: The study highlights the need for more nuanced evaluation, especially for abstractive summaries, where current automatic metrics may not fully capture factual correctness or semantic equivalence.
Key Results
- Gigaword: Found to be very noisy (44.5% of samples) and predominantly extractive (24.5%).
- XSum: Highly abstractive with no extractive samples, but also the noisiest (55% of samples).
- CNN/DM: Cleaner than the other two, with 64% of samples being paraphrase or inference, and only 18% exhibiting data noise.
- Model Performance Varies by Class: Models generally perform better on simpler (extractive) samples than on more complex (paraphrase, inference) ones, across all metrics.
- Metric Reliability Decreases with Complexity: For CNN/DM, the correlation between automatic metrics and human judgment decreases as summarization complexity increases. For instance, ROUGE-1 and MoverScore correlation with human judgment is higher for Extractive samples (around 0.82) than for Paraphrase/Inference samples (around 0.72).
Limitations and Future Work
The authors acknowledge several limitations:
- Single Label Typology: Forcing samples into a single category might oversimplify their nature. Future work could explore a dual-label system for quality and complexity.
- Positional Bias: The impact of positional bias in summaries, explored in prior work, was not directly included in this typology and is a direction for future research.
- Reference Quality: The study points out that web-scraped references can have quality issues, suggesting the need for more diverse reference sets and potentially incorporating automatic factuality metrics into human judgment procedures.
- Architecture-Specific Performance: The current analysis focused on averaged trends. Future work could investigate how specific model architectures perform across different sample classes.
The paper concludes by emphasizing the critical role of dataset characteristics in understanding model performance and metric reliability, advocating for more informed dataset creation and evaluation practices.
Summary updated