Abstract
Neural extractive summarization models have achieved significant performance gains, yet their ability to generalize across domains remains limited. This paper shifts the focus from architectural optimization to understanding dataset-inherent biases. By quantifying factors such as sentence position and content overlap, the authors reveal that models frequently rely on dataset-specific shortcuts rather than semantic understanding. They propose a framework to measure these biases and demonstrate that simple, data-aware training adjustments can outperform complex architectures and achieve state-of-the-art results.
Introduction
In neural extractive summarization—the task of selecting the most salient sentences from a document—performance has historically been measured by ROUGE scores on specific benchmarks. However, models often fail when transferred between domains, such as from news articles to scientific papers. This research argues that the field has overlooked the influence of dataset properties on model generalization. Instead of asking how to build a "better" model, the authors ask: what specific biases in the training data are the models actually learning?
Quantifying Dataset Bias
The researchers propose a systematic framework to quantify dataset characteristics, categorized into Constituent Factors and Style Factors.
Constituent factors include the Positional Value (P-Value), which measures where summary-worthy sentences typically appear, and the Content Value (C-Value), which identifies semantic patterns frequent in ground-truth summaries. To compare datasets, they introduce the Positional Coverage Rate (PCR) and Content Coverage Rate (CCR). Style factors focus on Density (the degree of direct copying) and Compression (the length ratio between source and summary).
Experimental Setup
The study evaluates six mainstream datasets to capture diverse domain characteristics:
- News: CNN/DM, Newsroom, NYT50, and DUC2002.
- Science: arXiv and PubMed.
By comparing news (characterized by heavy "lead bias") to scientific papers (where important information is distributed throughout the text), the authors highlight the structural differences that challenge model transferability.
The Impact of Positional and Content Bias
A primary finding is the "Lead Bias Trap." Models trained on news datasets learn to favor the beginning of documents. When these models encounter scientific papers where salient information is more distributed, performance drops significantly.
As shown in the experimental results, model accuracy is highly sensitive to these metrics. Accuracy tends to decrease as the positional distribution becomes more complex (higher P-Value) and increases when content patterns are more distinct (higher C-Value). Interestingly, while BERT improves overall performance, it primarily excels at identifying content patterns rather than overcoming positional distribution shifts.
Key Insights and Results
- Simple Tags, Big Gains: One of the most significant findings is that adding a simple "domain tag" to training samples allows models to learn domain-specific priors. This simple adjustment improved a standard Transformer's ROUGE-1 score on CNN/DM from 41.31 to 41.41, and to 42.77 when combined with BERT, outperforming previous state-of-the-art models.
- Architecture vs. Data: LSTMs were found to handle "hard" cases (low density/high compression) slightly better than Transformers, suggesting that newer architectures are not universally superior for all data subsets.
- Cross-Dataset Correlation: There is a direct correlation between PCR/CCR scores and cross-domain performance. Low overlap in positional or content factors between two datasets reliably predicts poor model transferability.
Conclusion
This study demonstrates that mining the characteristics of a dataset is as crucial as model design. By accounting for positional and content biases, researchers can build more robust summarization systems. Future work should extend this analysis to abstractive summarization and develop more sophisticated methods for handling high-compression "hard cases" where current models still struggle.
Summary updated