Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval

Jinrui Yang, Timothy Baldwin, Trevor Cohn

Introduction

In the evolving landscape of Information Retrieval (IR), the transition from keyword matching to sophisticated supervised models has expanded the capabilities of search engines. However, as retrieval systems become increasingly global, two critical gaps have emerged: the "True" Multilingual Gap and the Fairness Gap. Most existing benchmarks, such as Mr. TYDI, primarily evaluate monolingual retrieval across various languages—testing how well a system finds a Swedish document using a Swedish query. They rarely address "one-vs-many" scenarios, where a single query must retrieve relevant documents from a mixed-language pool. Furthermore, IR research often lacks datasets that link documents to the protected demographic attributes of their authors, making it difficult to measure and mitigate algorithmic bias.

To address these challenges, we introduce Multi-EuP, a new multilingual benchmark dataset sourced from European Parliament debates. Multi-EuP provides a robust framework for analyzing both language bias and demographic fairness in high-stakes, real-world multilingual IR.

The Multi-EuP Dataset

Multi-EuP is distinguished by its scale, quality, and rich metadata. Sourced from authentic speeches delivered in the European Parliament from 2020 to the present, the dataset includes:

  • Scale: 22,000 documents and 1,100 unique topics spanning 24 official EU languages.
  • High-Quality Translations: Unlike datasets like mMARCO, which rely on machine translation, Multi-EuP features topics professionally translated by the European Union, eliminating the confounding errors associated with automated translation.
  • Demographic Metadata: The dataset includes gender, nationality, political affiliation, and age for 705 Members of the European Parliament (MEPs). This allows researchers to move beyond language bias and investigate how retrieval algorithms treat different demographic groups.
  • Judgment Density: Because speeches are directly linked to specific debate topics, the relevance judgments are binary and comprehensive, providing a clearer signal for model training and evaluation compared to sparser datasets.

Methodology

The researchers established a baseline for Multi-EuP using BM25, a standard statistical retrieval algorithm. The study explored two primary experimental configurations:

  1. One-vs-One: Traditional monolingual or cross-lingual retrieval (e.g., English query to English documents).
  2. One-vs-Many: A single-language query searching against a comprehensive multilingual document collection containing all 24 languages.

A central technical focus was the impact of tokenization strategies. The study compared sophisticated, language-specific tokenizers—designed to handle the linguistic nuances of individual languages—against a simple "whitespace" tokenizer to determine how these choices influence language bias in a multilingual ranking context.

Key Insights and Results

The Tokenization Paradox

A significant finding of the study is that language-specific tokenizers, while improving performance for individual languages in isolation, actually increase language bias in multilingual retrieval. These tokenizers create a stronger alignment between the query and documents in the same language, causing the model to favor the query's native language even more aggressively. Conversely, using a simple whitespace tokenizer for both indexing and queries acted as an equalizer. Although it slightly lowered overall performance, it reduced language bias and created a more level playing field across the 24 languages.

Language-Specific Challenges

The analysis identified Polish as a particularly "BM25-unfriendly" language. Due to its highly inflected nature—where word forms change significantly based on grammar—simple lexical matching often fails to capture relevance, resulting in lower scores compared to morphologically simpler languages. Additionally, the results showed strong correlations within language families; for instance, an Italian query (Romance family) is more likely to surface relevant documents in Spanish than in a Slavic language like Polish.

Retrieval Performance

Multi-EuP serves as a relatively accessible benchmark for monolingual IR, with an average Mean Reciprocal Rank (MRR@100) of 49.61% across all languages. In the one-vs-many setup, English queries demonstrated the highest performance (MRR 62.79%), largely because English accounts for the highest volume of documents and relevance judgments in the collection.

Limitations and Future Work

While Multi-EuP provides a significant step forward, the authors acknowledge several constraints. The data is limited to political discourse from the last three years, which may not generalize to specialized fields like medical or legal IR. Furthermore, while the dataset contains rich demographic metadata, this initial study focused primarily on language bias. Future research is required to explore demographic fairness (gender, age, etc.) and to determine if modern neural re-rankers, such as mDPR or ColBERT, exhibit the same biases observed in the BM25 baseline.

Conclusion

Multi-EuP offers a unique resource for the IR community, enabling the study of retrieval systems in a truly multilingual and demographically diverse context. By providing high-quality, professionally translated data and rich metadata, it paves the way for developing search engines that are not only more effective across languages but also fairer to the diverse authors they index.

Summary updated

Abstract

We present Multi-EuP, a new multilingual benchmark dataset, comprising 22K multilingual documents collected from the European Parliament, spanning 24 languages.This dataset is designed to investigate fairness in a multilingual information retrieval (IR) context to analyze both language and demographic bias in a ranking context.It boasts an authentic multilingual corpus, featuring topics translated into all 24 languages, as well as cross-lingual relevance judgments.Furthermore, it offers rich demographic information associated with its documents, facilitating the study of demographic bias.We report the effectiveness of Multi-EuP for benchmarking both monolingual and multilingual IR.We also conduct a preliminary experiment on language bias caused by the choice of tokenization strategy.
BibTeX
@misc{yang2023multi,
  author = {Jinrui Yang and Timothy Baldwin and Trevor Cohn},
  title = {{Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval}},
  year = {2023},
  eprint = {2311.01870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.01870},
}
Figure