Abstract
This paper introduces the Multilingual Model Selection Problem (MMSP): the challenge of choosing the "best" language model when performance varies significantly across hundreds of languages. We argue that the standard practice of reporting average accuracy is a biased "utilitarian" choice and propose Rawlsian fairness—maximizing the performance of the worst-off language—as a more robust and ethically sound framework for evaluating multilingual AI.
Introduction
We are currently in the era of Massively Multilingual Pre-trained Language Models (MultiLMs) like mBERT and XLM-RoBERTa. These models are celebrated for "zero-shot transfer," where a model trained on English can suddenly perform tasks in languages like Swahili or Quechua. However, a major evaluation gap exists: if Model A is better than Model B on 10 languages, but Model B has a higher overall average accuracy, which one is truly "better"?
The Multilingual Model Selection Problem (MMSP)
Most researchers default to average accuracy, a utilitarian approach. This is problematic because the distribution of digital resources is extremely skewed. Relying on averages often masks poor performance in low-resource languages, further widening the "digital divide." The MMSP seeks a formal, principled way to make model selection choices that are linguistically fair, drawing on social choice theory and economics.
Normative Principles for Evaluation
The authors evaluate the MMSP through three primary lenses:
- Pareto-Efficiency: A model is superior if it is at least as good as another in all languages and strictly better in at least one. While ideal, this rarely occurs in practice.
- Utilitarianism (The Status Quo): Choosing the model with the highest average performance. This assumes that high performance in one language can compensate for failure in another.
- Rawlsian Fairness (The Proposal): Based on John Rawls’ "Difference Principle," this suggests choosing the model that maximizes the minimum performance (max-min). In other words, the best model is the one that performs the best on its weakest language.
Key Results and Empirical Re-analysis
The authors re-analyzed results from several major multilingual benchmarks to see how different fairness lenses change the "winner."
XGLUE Re-analysis
While Unicoder is often the winner by average, the Rawlsian analysis confirms it is also the "fairest" choice for 7 out of 9 tasks. It consistently raised the performance floor for the worst-performing languages across various tasks like NER and Question Answering.
XTREME Re-analysis
The results here were more nuanced. While XLM-R wins on average, a Rawlsian lens shows that MMTE (Massively Multilingual Translation Encoder) is the superior choice for tasks like Part-of-Speech (POS) tagging and Named Entity Recognition (NER) because its "minimum" performance is significantly higher than XLM-R’s.
Key Insights
- The Average is a Choice: Reporting average accuracy isn't a neutral scientific fact; it’s an implicit commitment to utilitarianism.
- Rawlsian is Robust: The Rawlsian "max-min" choice is mathematically equivalent to a "robust utilitarian" choice under uncertainty. If you don't know which language will be most important to your users tomorrow, protecting the worst-performing language today is the safest bet.
- The "Curse of Multilinguality": As models try to support more languages, their average performance often peaks and then declines. A fairness-based approach helps navigate this trade-off more ethically.
Limitations and Future Directions
The authors treat "languages" as the primary units of fairness, though they acknowledge that fairness should ideally apply to users. However, measuring user-level utility across 100+ languages is currently a data impossibility. They suggest that future leaderboards (like XTREME) should offer alternative rankings based on max-min or "lex-min" (maximizing the minimum, then the second-minimum, etc.) to encourage researchers to focus on low-resource language stability.
Summary updated