TL;DR
This paper introduces XL-Sum, a massive new dataset of 1 million article-summary pairs across 44 languages, addressing the critical lack of resources for multilingual abstractive text summarization. By fine-tuning the mT5 model on XL-Sum, they demonstrate competitive summarization performance across many languages, including low-resource ones, paving the way for more equitable NLP research.
What Problem Does This Paper Address?
Automatic text summarization aims to condense long documents into shorter, informative pieces. While extractive summarization (selecting and concatenating existing sentences) has been around for a while, abstractive summarization (generating new sentences, potentially with paraphrasing) offers more coherent and concise summaries. However, abstractive summarization, especially for languages other than English, has been severely hampered by the scarcity of large, high-quality datasets. Most existing datasets are English-centric, leaving a significant gap in research and development for the vast majority of the world's languages. This lack of data makes it difficult to train robust multilingual models and hinders progress in making advanced NLP technologies accessible to a wider global community.
Methods
The core of this work is the creation of the XL-Sum dataset. The authors developed a custom web crawler to extract article-summary pairs from the BBC's news website. They employed a set of heuristics to identify and extract summaries, focusing on bolded paragraphs at the beginning of articles that contained the main story, rather than just headlines. This approach aimed to capture professionally written, abstractive summaries.
To evaluate the dataset and demonstrate its utility, they fine-tuned mT5, a state-of-the-art pre-trained multilingual Transformer model. They conducted experiments in two settings:
- Multilingual Summarization: Training a single mT5 model on data from multiple languages simultaneously.
- Low-Resource Summarization: Training mT5 models on individual low-resource languages to assess their performance in isolation.
They also performed human evaluations and intrinsic evaluations (using metrics like novel n-gram ratio, abstractivity, compression, and redundancy) to assess the quality of the generated summaries.
Datasets
The primary contribution is the XL-Sum dataset, which contains approximately 1 million professionally annotated article-summary pairs from BBC news articles. This dataset covers 44 languages, including many low-resource languages for which no public abstractive summarization datasets previously existed. The authors also used existing datasets like CNN/DM and XSum for comparison in their intrinsic evaluation.
Key Insights and Ideas
- The Power of a Single Source: Collecting data from a single source (BBC) across many languages ensures a consistent summarization style, which is crucial for effective multilingual model training. This contrasts with datasets where samples are collected from diverse sources, potentially leading to varied summarization strategies.
- Bridging the Low-Resource Gap: XL-Sum significantly expands the availability of data for low-resource languages, democratizing access to advanced NLP capabilities and enabling research that was previously infeasible.
- Multilingual Transfer Learning is Effective: Training a single model on multiple languages (multilingual training) shows strong performance, suggesting that languages can benefit from shared linguistic structures and patterns, even across different families.
- Human-Written Summaries Contain "External" Knowledge: Human evaluators noted that summaries sometimes contained information not directly present in the article but inferable from common sense or domain knowledge. This highlights a challenge for models that are strictly trained on input text alone.
Key Results
- Dataset Size: XL-Sum is the largest abstractive summarization dataset in terms of both the number of samples (1 million) and the number of languages covered (44).
- Summarization Quality: Human evaluations indicated high quality, with summaries generally conveying the article's main idea (Property A >90% for most languages) and showing low inconsistency (Property B <1% for most languages). Property C (information not directly inferable) was higher, suggesting a common characteristic of human-written summaries.
- Intrinsic Evaluation: XL-Sum summaries exhibit high abstractiveness and conciseness, outperforming CNN/DM on key metrics and being comparable to XSum (though XSum is English-only).
- Model Performance: Fine-tuning mT5 on XL-Sum achieved competitive results. For multilingual summarization, they reported ROUGE-2 scores higher than 11 on 10 languages, with some exceeding 15. Even when trained on individual low-resource languages, the models performed competitively, with differences of no more than 2 ROUGE-2 points compared to the multilingual baseline.
Limitations and Future Work
- Limitations:
- The authors acknowledge that some negative summaries in human evaluation were extracted from opinion pieces or blog posts, which may not always reflect the main story.
- The presence of "external" information in summaries (Property C) is a characteristic of human summarization that poses a challenge for models aiming for strict faithfulness to the input text.
- For some low-resource languages like Amharic and Azerbaijani, performance was lower, potentially due to smaller training sample sizes and the lack of reliable stemmers for evaluation.
- Future Work:
- The authors plan to investigate the use of XL-Sum for other summarization tasks, such as cross-lingual summarization.
- They also suggest exploring how incorporating real-world knowledge and commonsense reasoning into language models could further improve summarization performance, given the observed "external" information in human summaries.
- The dataset curation tool is made available, encouraging the community to expand the dataset further over time.
Summary updated