Factual Consistency of Multilingual Pretrained Language Models

Constanza Fierro, Anders Søgaard

Introduction

Multilingual pretrained language models (PLMs) such as mBERT and XLM-R are frequently treated as "soft" knowledge bases. Researchers and developers query these models to retrieve facts, such as "Where was Albert Einstein born?". However, for a model to serve as a reliable knowledge source, it must demonstrate consistency. A robust model should provide the same answer regardless of whether it is asked "Einstein was born in [MASK]" or "The birthplace of Einstein was [MASK]."

While previous studies have highlighted consistency issues in English-only models, there has been a significant gap in our understanding of how multilingual models behave across different languages. This paper introduces mPARAREL, a large-scale dataset covering 46 languages, to show that these models are surprisingly "brittle" and inconsistent, particularly in non-English contexts.

The mPARAREL Dataset

The centerpiece of this research is mPARAREL, a multilingual expansion of the English PARAREL dataset. It encompasses 38 different relations (e.g., "born-in," "native-language") across 46 languages.

mPARAREL dataset statistics.

To construct the dataset, the authors utilized five machine translation models to translate English templates. To ensure high quality, human reviews were conducted for 11 languages, including Greek, German, Spanish, and Japanese. Native speakers corrected or removed poor translations, providing a gold-standard subset for validation.

Methodology

The authors employ a "cloze-style" probing framework. For a given fact (a subject-relation-object triple from Wikidata), the model is queried using various paraphrased templates. Because many objects (like city names) consist of multiple tokens, the probability of an object is calculated by averaging the probabilities of its constituent tokens.

The study evaluates models using three core metrics:

  1. Consistency: The frequency with which a model provides the same top prediction for different paraphrases of the same fact.
  2. Accuracy: The frequency with which the model's top prediction is correct.
  3. Consistency-Accuracy: A stricter metric measuring how often the model is both correct and consistent across all paraphrases of a fact.

Key Insights and Results

The experimental results reveal several critical weaknesses in current multilingual PLMs.

The "English Peak" and Cross-Lingual Drop

Multilingual models exhibit their highest consistency in English. Consistency scores drop sharply for the other 45 languages. For example, while mBERT achieves approximately 37% accuracy in English, this figure plummets to ~13% in Japanese and ~22% in Chinese. This suggests that the "world knowledge" captured by these models is heavily biased toward English-centric linguistic patterns.

Consistency vs. Accuracy

A key finding is that high consistency does not imply high accuracy. In Japanese, mBERT showed relatively high consistency but low consistency-accuracy. This occurs because the model often predicts the same frequent, yet incorrect, token regardless of the specific paraphrase used.

Consistency, Accuracy, and Consistency-accuracy across 46 languages.

Model Fairness and Brittleness

While mBERT and XLM-R show similar average consistency, XLM-R demonstrates a lower standard deviation across languages, suggesting it is more "fair" in its performance distribution. However, both models remain extremely brittle; even minor changes, such as adding a period at the end of a query, can trigger a change in the model's prediction. Furthermore, manual corrections in the "reviewed" data subsets generally improved consistency and accuracy scores by 1-2%, highlighting the importance of high-quality benchmarks.

Conclusion and Future Work

The introduction of mPARAREL highlights the significant consistency gap in multilingual PLMs. The study demonstrates that these models are not yet reliable as universal knowledge bases. Future research should focus on developing techniques, such as specialized adapters or consistency-enforcing fine-tuning, to ensure that a model's underlying "belief" about a fact remains stable across different languages and phrasings.

Summary updated

Abstract

Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-the-blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, mParaRel, and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.
BibTeX
@inproceedings{fierro2022factual,
  author = {Constanza Fierro and Anders Søgaard},
  title = {{Factual Consistency of Multilingual Pretrained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=woP8vvtH_s},
}
Figure