Introduction
Multilingual pretrained language models (PLMs) such as mBERT and XLM-R are frequently treated as "soft" knowledge bases. Researchers and developers query these models to retrieve facts, such as "Where was Albert Einstein born?". However, for a model to serve as a reliable knowledge source, it must demonstrate consistency. A robust model should provide the same answer regardless of whether it is asked "Einstein was born in [MASK]" or "The birthplace of Einstein was [MASK]."
While previous studies have highlighted consistency issues in English-only models, there has been a significant gap in our understanding of how multilingual models behave across different languages. This paper introduces mPARAREL, a large-scale dataset covering 46 languages, to show that these models are surprisingly "brittle" and inconsistent, particularly in non-English contexts.
The mPARAREL Dataset
The centerpiece of this research is mPARAREL, a multilingual expansion of the English PARAREL dataset. It encompasses 38 different relations (e.g., "born-in," "native-language") across 46 languages.
To construct the dataset, the authors utilized five machine translation models to translate English templates. To ensure high quality, human reviews were conducted for 11 languages, including Greek, German, Spanish, and Japanese. Native speakers corrected or removed poor translations, providing a gold-standard subset for validation.
Methodology
The authors employ a "cloze-style" probing framework. For a given fact (a subject-relation-object triple from Wikidata), the model is queried using various paraphrased templates. Because many objects (like city names) consist of multiple tokens, the probability of an object is calculated by averaging the probabilities of its constituent tokens.
The study evaluates models using three core metrics:
- Consistency: The frequency with which a model provides the same top prediction for different paraphrases of the same fact.
- Accuracy: The frequency with which the model's top prediction is correct.
- Consistency-Accuracy: A stricter metric measuring how often the model is both correct and consistent across all paraphrases of a fact.
Key Insights and Results
The experimental results reveal several critical weaknesses in current multilingual PLMs.
The "English Peak" and Cross-Lingual Drop
Multilingual models exhibit their highest consistency in English. Consistency scores drop sharply for the other 45 languages. For example, while mBERT achieves approximately 37% accuracy in English, this figure plummets to ~13% in Japanese and ~22% in Chinese. This suggests that the "world knowledge" captured by these models is heavily biased toward English-centric linguistic patterns.
Consistency vs. Accuracy
A key finding is that high consistency does not imply high accuracy. In Japanese, mBERT showed relatively high consistency but low consistency-accuracy. This occurs because the model often predicts the same frequent, yet incorrect, token regardless of the specific paraphrase used.
Model Fairness and Brittleness
While mBERT and XLM-R show similar average consistency, XLM-R demonstrates a lower standard deviation across languages, suggesting it is more "fair" in its performance distribution. However, both models remain extremely brittle; even minor changes, such as adding a period at the end of a query, can trigger a change in the model's prediction. Furthermore, manual corrections in the "reviewed" data subsets generally improved consistency and accuracy scores by 1-2%, highlighting the importance of high-quality benchmarks.
Conclusion and Future Work
The introduction of mPARAREL highlights the significant consistency gap in multilingual PLMs. The study demonstrates that these models are not yet reliable as universal knowledge bases. Future research should focus on developing techniques, such as specialized adapters or consistency-enforcing fine-tuning, to ensure that a model's underlying "belief" about a fact remains stable across different languages and phrasings.
Summary updated