MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation

Andreea Iana, Goran Glavas, Heiko PaulheimSIGIR 2024 · arXiv

Introduction

News recommendation research has historically operated within a "monolingual bubble," primarily focusing on English-language content and users. This English-centric approach neglects billions of polyglot users and speakers of low-resource languages. To bridge this gap, researchers have introduced xMIND, the first large-scale, open-source multilingual dataset for news recommendation, covering 14 diverse languages.

The xMIND Dataset and Methodology

The authors constructed xMIND by translating 130,379 unique news articles from the industry-standard Microsoft News (MIND) dataset into 14 languages. This process utilized the NLLB (No Language Left Behind) 3.3B model, an open-source machine translation system optimized for diverse and low-resource languages. The resulting dataset spans 13 language families and 6 scripts, including high-resource languages like Mandarin (CMN) and Japanese (JPN), as well as low-resource ones like Somali (SOM), Guarani (GRN), and Haitian Creole (HAT).

To evaluate the dataset, seven state-of-the-art Neural News Recommenders (NNRs)—including NAML, LSTUR, and TANR—were benchmarked. The standard news encoders in these models were replaced with a multilingual pre-trained language model, XLM-RoBERTa. Evaluation focused on two primary scenarios:

  • Zero-Shot (ZS-XLT): Models trained on English and tested on a target language without prior exposure.
  • Few-Shot (FS-XLT): Models trained on a mix of English and small amounts of target-language data.

Key Results and Language Variance

Benchmarking revealed a significant performance drop in cross-lingual settings. In zero-shot scenarios, models experienced an average relative decline in ranking performance (nDCG@10) between 3.78% and 8.11% compared to English-only baselines.

Model performance (nDCG@10) across 14 languages in the xMIND dataset.

Performance varied drastically by language. As seen in the figure above, models performed best on languages well-represented in AI pre-training (e.g., Finnish, Romanian) but struggled with those involving unique scripts or fewer resources (e.g., Georgian, Thai). Languages not seen during the pre-training of the underlying XLM-RoBERTa model, such as Guarani or Haitian Creole, saw the most severe performance crashes.

The "Polyglot Penalty" and User Encoding

A critical finding of the study is the "Polyglot Penalty." Current user encoders—the components responsible for modeling user interests—become "confused" when a user's consumption history is bilingual. Even when the underlying topics are identical, the model struggles to map interests across linguistic boundaries.

The "Polyglot Penalty": Performance changes based on the percentage of target language in bilingual user consumption.

As shown in the heatmap above, performance generally degrades as the percentage of the target language in a bilingual user's history increases during testing. This suggests that existing architectures fail to create truly language-agnostic interest profiles, creating a hurdle for polyglot users who consume news in multiple languages.

Additional Insights

  • The 30% Threshold: In few-shot learning, adding small amounts of local language data can initially hurt performance. A model typically requires at least 30-40% of its training data to be in the target language before it consistently outperforms a purely English-trained model.
  • Robustness to Translation: Interestingly, the bottleneck is not machine translation quality. Models performed similarly on NLLB translations and Google's commercial GNMT translations, suggesting the "signal" for user interest is robust to minor linguistic artifacts.

Limitations and Future Directions

The authors identify several constraints in the current work. First, because xMIND is a translation of a US-based dataset, the content may lack cultural relevance for speakers in other regions. Second, the relatively short user histories (averaging 33 articles) make it difficult to learn stable cross-lingual interest profiles. Finally, the dataset remains synthetic.

Future research should focus on developing more robust, language-agnostic user encoders and gathering "born-multilingual" datasets with organic click logs from diverse global regions.

Summary updated

BibTeX
@inproceedings{iana2024mind,
  author = {Andreea Iana and Goran Glavas and Heiko Paulheim},
  title = {{MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
}
Figure