Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning

Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, Xiang Ren

TL;DR

While large language models (LLMs) show impressive common sense in English, they often stumble when asked to reason in other languages. This paper introduces Mickey Probe, a language-agnostic framework to evaluate multilingual common sense, and Multilingual Contrastive Pre-training (MCP), a simple yet powerful method that significantly boosts a model's ability to reason across 15 different languages.

The Challenge of Multilingual Common Sense

Commonsense reasoning (CSR)—the ability to make inferences based on general facts about the physical world and human behavior—is the "implicit background" of human communication. However, CSR research has been overwhelmingly English-centric.

Transitioning from English-centric evaluation to a multilingual commonsense framework.

Existing evaluation methods, like the LAMA probe, are limited because they only test a model's ability to recall a single masked word (e.g., "Birds have [mask]"). This doesn't work well for many languages where concepts are represented by multiple tokens, nor does it account for the fact that multiple words could be "correct" (e.g., "wings," "feathers," or "beaks"). Furthermore, there has been a lack of high-quality, large-scale multilingual benchmarks to test if a model trained on English data can transfer its "common sense" to a zero-shot setting in languages like Swahili or Urdu.

MickeyProbe and Multilingual Contrastive Pre-training

The authors tackle these gaps with three main contributions:

  1. Mickey Probe: Instead of predicting a single word, this task asks a model to rank a set of full sentences (assertions) based on their plausibility. It uses Pseudo-Log-Likelihood (PLL) to score sentences, which allows it to evaluate masked language models on whole-sentence semantics rather than just local word patterns.
The MickeyProbe framework uses Pseudo-Log-Likelihood (PLL) to rank assertions across multiple languages.
  1. Mickey Corpus: To power the probe, they created a massive dataset of 561k sentences across 11 languages. They started with English commonsense facts, used "BERT-attacks" to create plausible-sounding but incorrect distractors, and then used bidirectional translation with a "BT-Cosine" filter to ensure high quality.
  2. Multilingual Contrastive Pre-training (MCP): This is a new "pretext task" designed to improve sentence-level representations. During training, the model is given a set of assertions in different languages and must identify the one that is actually true. This forces the model to align commonsense concepts across its entire multilingual embedding space.

Datasets and Benchmarks

The research utilizes several key datasets:

  • Mickey Corpus: 561k sentences in 11 languages (EN, DE, IT, ES, FR, NL, RU, BG, VI, ZH, HI) used for probing and MCP.
  • X-CSQA & X-CODAH: New multilingual versions of the popular CommonsenseQA and CODAH benchmarks. These were created by translating the English originals into 15 other languages, including low-resource languages like Swahili (sw) and Urdu (ur).

Key Insights

  • Model "Personalities": Different multilingual models have distinct language preferences. For example, while XLM-R generally performs best, mBERT and d-mBERT actually show stronger performance in French than in German or Dutch, despite having similar training objectives.
  • Data Size vs. Quality: Interestingly, the size of the pre-training corpus for a specific language doesn't always correlate with commonsense performance. Russian (ru) has a massive corpus in the CC-100 dataset, yet models often performed worse in Russian than in languages with much smaller training sets, like German. This suggests that vocabulary design and text quality are more critical than raw volume.
  • The Power of Contrast: The MCP method proved that "common sense" is largely language-invariant. By training the model to distinguish truth from nonsense across a mix of languages, the authors found that the model's performance improved even on languages it didn't see during the MCP phase.

Results and Performance

  • Performance Boost: Applying MCP to the state-of-the-art XLM-R model yielded an average accuracy gain of +2.7% on X-CSQA and +2.3% on X-CODAH across all tested languages.
  • Efficiency: Models enhanced with MCP were much more "sample efficient," reaching higher accuracy levels faster during fine-tuning than vanilla models.
  • Zero-Shot Transfer: While English-only RoBERTa still leads on English tests, the gap is closing. For X-CSQA, MCP-enhanced XLM-R reached 69.5% accuracy, nearing RoBERTa’s 70.4%.

Limitations and Future Work

The authors acknowledge the issue of cultural bias. Common sense isn't always universal; for instance, the "usual" time for a wedding varies by culture. While they manually filtered out highly controversial or culturally specific examples, they admit that a truly "global" common sense is difficult to define. Future work could involve moving beyond automatic translation to human-authored multilingual datasets and exploring how these improved representations affect downstream tasks like dialogue systems.

Summary updated

Abstract

Commonsense reasoning research has so far been limited to English. We aim to evaluate and improve popular multilingual language models (ML-LMs) to help advance commonsense reasoning (CSR) beyond English. We collect the Mickey Corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and improving ML-LMs. We propose Mickey Probe, a language-agnostic probing task for fairly evaluating the common sense of popular ML-LMs across different languages. In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 15 other languages, so that we can evaluate popular ML-LMs for cross-lingual commonsense reasoning. To improve the performance beyond English, we propose a simple yet effective method -- multilingual contrastive pre-training (MCP). It significantly enhances sentence representations, yielding a large performance gain on both benchmarks.
BibTeX
@inproceedings{lin2021common,
  author = {Bill Yuchen Lin and Seyeon Lee and Xiaoyang Qiao and Xiang Ren},
  title = {{Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.102},
}
Figure