Abstract
Pretrained language models (PLMs) are increasingly utilized as repositories of factual knowledge for downstream tasks. However, the reliability of this knowledge depends on consistency—the ability of a model to provide the same answer regardless of how a query is phrased. While previous research has highlighted consistency issues in English-only models, this study extends the investigation to a multilingual context. We introduce mParaRel, a comprehensive benchmark covering 46 languages, to evaluate models like mBERT and XLM-R. Our findings reveal that while these models are most consistent in English, their reliability significantly degrades in other languages, uncovering a critical gap in the robustness of multilingual AI.
Introduction: The Neural Database Problem
Modern Pretrained Language Models (PLMs) are often treated as "neural databases." When we ask a model a question like "Where was Albert Einstein born?", we expect it to retrieve a fact stored during its training. For a database to be truly useful, it must be consistent. If the query is slightly rephrased to "What is Einstein's place of birth?", a reliable model should provide the same answer.
Previous studies have shown that English-centric models are "fickle," frequently changing their predictions based on minor linguistic variations. This research explores whether multilingual models—designed to handle dozens of languages simultaneously—suffer from the same instability, and whether this inconsistency is uniform across different linguistic families.
The mParaRel Benchmark
To investigate these questions, we developed mParaRel, a multilingual extension of the English PARAREL dataset.
- Scope: The dataset covers 38 distinct factual relations (e.g., "place of birth," "native language," "employer") across 46 languages.
- Construction: Templates were generated using five machine translation models. To ensure the highest quality, native speakers reviewed and corrected translations for 11 of the languages, involving 14 contributors.
- Content: The benchmark utilizes thousands of subject-relation-object triples sourced from Wikidata, providing a robust foundation for testing factual retrieval across diverse linguistic structures.
Methodology: Probing for Consistency
The study employs a cloze-style probing framework. Models are presented with a sentence containing a masked token (e.g., "[MASK] is the capital of France") and tasked with filling in the blank.
Consistency is measured by comparing the model's top predictions across sets of paraphrased templates for the same fact. If a model is asked five different ways about a person's birthplace, the consistency score reflects the percentage of pairs that yield the identical top prediction.
A significant technical challenge in multilingual evaluation is the variation in tokenization. To address this, the researchers used a multi-token prediction approach. Instead of requiring a single-token match, they averaged the probabilities of the tokens comprising an object's name (e.g., "New York"). This ensures that models are not unfairly penalized for languages with complex morphological structures or longer word forms.
Key Insights: Accuracy vs. Consistency
One of the paper's most critical findings is the Consistency-Accuracy Trap. A model can be highly consistent but consistently wrong.
- The Japanese Case: In Japanese, mBERT showed surprisingly high consistency. However, further analysis revealed that the model was often "lazily" predicting the same high-frequency token regardless of the specific query.
- Consistency-Accuracy Metric: To account for this, the authors emphasize "consistency-accuracy"—the subset of predictions that are both stable across paraphrases and factually correct. High consistency is only valuable if it aligns with factual truth.
Cross-Lingual Performance and Fairness
The evaluation of mBERT and XLM-R revealed several distinct trends:
- English Dominance: Both models are most consistent when queried in English. mBERT’s consistency in English (~0.54) is nearly identical to its monolingual BERT counterpart, suggesting that multilingual training does not inherently solve the consistency problem.
- The Multilingual Drop: Performance drops significantly for almost all other languages. While English and Vietnamese showed higher stability, most other languages lagged behind considerably.
- Model Comparison: While mBERT and XLM-R share similar average consistency scores (~0.43), XLM-R exhibits a lower standard deviation across languages. This suggests that XLM-R is "fairer," providing a more uniform (though still imperfect) level of reliability across the globe compared to mBERT's more erratic performance.
The Brittleness of Prompting
The research also highlights how sensitive these models are to "surface-level" changes. For instance, the mere presence or absence of a period at the end of a prompt can alter consistency by up to 5% in languages like Chinese or Japanese. Furthermore, manual correction of the dataset only improved scores by 1-2%, indicating that the observed inconsistency is not a result of "noisy" data, but rather a fundamental limitation in how these models represent and retrieve knowledge.
Conclusion and Future Directions
The introduction of mParaRel demonstrates that current multilingual PLMs are remarkably brittle when it comes to factual consistency. While they can act as knowledge bases, their "fickleness" in non-English contexts poses a significant barrier to reliable deployment in global applications.
Future work should focus on making these models more robust to irrelevant input variations. Potential solutions include the use of adapters or prompt optimization techniques to stabilize factual retrieval. As AI continues to scale globally, ensuring that a model is as reliable in Swahili or Japanese as it is in English remains a vital challenge for the research community.
Summary updated