Abstract: Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Despite a surge in neural architectures for text summarization, the field has reached a point of stagnation. While new models are released frequently, their performance gains over simple baselines, such as Lead-3 (selecting the first three sentences of an article), are marginal. This research identifies a "vicious cycle" within the research ecosystem that hinders genuine progress, driven by underconstrained datasets, flawed evaluation metrics, and inherent model biases.
The Problem of Layout Bias
A primary issue in current summarization research is the over-reliance on news datasets that follow the "Inverted Pyramid" structure, where the most critical information is concentrated at the beginning of the text. Models have learned to exploit this layout rather than developing a semantic understanding of the content. Analysis shows that approximately 60% of information considered important by humans is located in the first third of the article.
This bias creates a "Lead-3 trap." When models are evaluated against the Lead-3 baseline instead of ground-truth human references, their ROUGE scores often see a massive artificial jump (e.g., ROUGE-1 increasing from ~40 to ~69), proving that these models are largely mimicking a simple positional heuristic.
Flaws in Evaluation Metrics
The standard metric for the field, ROUGE (n-gram overlap), is fundamentally limited. It measures lexical similarity but fails to account for factual accuracy or semantic consistency. A model can achieve a high ROUGE score while generating a summary that is factually opposite to the source text (e.g., changing "no injuries" to "multiple injuries").
As shown in the correlation analysis, the Pearson correlation between ROUGE and human-rated "consistency" (factuality) is as low as 0.07 for abstractive models. This indicates that ROUGE is virtually useless for determining if a summary is actually true. Furthermore, the study found a "multi-reference paradox": adding more reference summaries unexpectedly decreased the correlation between ROUGE and human judgment.
Data Quality and Model Convergence
The datasets commonly used, such as CNN/DailyMail and Newsroom, are often noisy, containing unparsed HTML and placeholder text. This noise pollutes the training signal. Additionally, an audit of 13 state-of-the-art models revealed high similarity between their outputs; these models often have higher n-gram overlap with each other than they do with human-written references.
Manual inspection of summaries from top-tier abstractive models revealed a startling statistic: 30% of generated summaries contained blatant factual errors. This level of hallucination remains a significant barrier to real-world application.
Conclusion and Future Directions
To move beyond the current plateau, the research community must shift its focus:
- Beyond N-grams: Develop metrics that evaluate semantic meaning and factual entailment.
- Curated Data: Move away from noisy, automatically scraped datasets toward curated data with specific constraints that define what makes a summary "important."
- Domain Diversity: Recognize that different domains (e.g., legal, medical, or conversational) require different, layout-agnostic approaches rather than a one-size-fits-all model for news.
Summary updated