XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, Rifat Shahriyar

Introduction

Automatic text summarization is a critical task in natural language processing, traditionally divided into extractive and abstractive methods. While extractive summarization selects existing sentences from a text, abstractive summarization generates novel sentences to paraphrase content, offering a more human-like and concise output. However, abstractive models require vast amounts of high-quality training data, which has historically been available almost exclusively for English. This "resource gap" has left many languages without public benchmarks for abstractive summarization.

XL-Sum addresses this gap by introducing a massive, high-quality dataset featuring over 1 million professionally annotated article-summary pairs across 44 languages, ranging from high-resource languages like French to low-resource ones like Amharic and Welsh.

The XL-Sum Dataset

The researchers curated the XL-Sum dataset by developing a custom crawler for the British Broadcasting Corporation (BBC). They utilized the BBC’s consistent editorial style, where the lead paragraph typically serves as a professionally written summary. To ensure high quality and "abstractivity," the team applied strict heuristics: summaries had to be within the first two paragraphs, articles had to be at least twice the length of the summary, and multimedia-heavy pages were excluded.

Distribution of samples across the 44 languages in the XL-Sum dataset.

The resulting corpus contains 1,005,292 pairs. As shown in the distribution table above, the dataset covers a diverse range of language families and resource levels, from English (301,444 samples) to Scottish Gaelic (1,101 samples).

Methodology and "Abstractivity"

The authors fine-tuned the mT5 (Multilingual T5) model, a state-of-the-art transformer. They explored two primary training settings: a single multilingual model trained on all 44 languages and individual models for specific languages.

A key characteristic of XL-Sum is its high degree of abstractivity. Compared to established benchmarks like CNN/Daily Mail, XL-Sum summaries contain significantly more novel words. Approximately one-third of the tokens and over 75% of the bigrams in the summaries do not appear in the source text, forcing the model to learn true paraphrasing rather than simple copying.

Key Results and Multilingual Synergy

The multilingual model demonstrated strong performance across the board. It achieved ROUGE-2 scores higher than 11 for all ten benchmarked languages. On English, the model's ROUGE-2 of 15.18 proved highly competitive with models trained on English-only datasets like XSum (16.58).

One of the most significant findings was the "multilingual synergy." Training on all languages simultaneously consistently outperformed training on individual languages, particularly for low-resource cases.

Comparison of ROUGE scores between low-resource (monolingual) and multilingual training settings.

As illustrated in the comparison table, languages like Bengali saw a ROUGE-2 jump from 9.50 in the low-resource setting to 11.43 in the multilingual setting. This suggests that the model learns the fundamental concept of summarization from high-resource data and successfully transfers that skill to lower-resource languages.

Limitations and Future Directions

While XL-Sum represents a major step forward, hardware constraints limited the study to the "base" version of mT5 (600M parameters) and a 512-token input window. Future work with larger models and longer contexts is expected to improve results further. Additionally, the authors noted that evaluation metrics like ROUGE may need refinement for non-spaced languages like Japanese, which showed unusually high scores due to segmentation differences.

The authors are releasing their curation tools to the community to encourage the continued growth of the dataset and the exploration of cross-lingual summarization.

Summary updated

BibTeX
@inproceedings{hasan2021xl,
  author = {Tahmid Hasan and Abhik Bhattacharjee and Md. Saiful Islam and Kazi Samin Mubasshir and Yuan-Fang Li and Yong-Bin Kang and M. Sohel Rahman and Rifat Shahriyar},
  title = {{XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.413},
}
Figure