TL;DR
While large language models (LLMs) show impressive common sense in English, they often stumble when asked to reason in other languages. This paper introduces Mickey Probe, a language-agnostic framework to evaluate multilingual common sense, and Multilingual Contrastive Pre-training (MCP), a simple yet powerful method that significantly boosts a model's ability to reason across 15 different languages.
The Challenge of Multilingual Common Sense
Commonsense reasoning (CSR)—the ability to make inferences based on general facts about the physical world and human behavior—is the "implicit background" of human communication. However, CSR research has been overwhelmingly English-centric.
Existing evaluation methods, like the LAMA probe, are limited because they only test a model's ability to recall a single masked word (e.g., "Birds have [mask]"). This doesn't work well for many languages where concepts are represented by multiple tokens, nor does it account for the fact that multiple words could be "correct" (e.g., "wings," "feathers," or "beaks"). Furthermore, there has been a lack of high-quality, large-scale multilingual benchmarks to test if a model trained on English data can transfer its "common sense" to a zero-shot setting in languages like Swahili or Urdu.
MickeyProbe and Multilingual Contrastive Pre-training
The authors tackle these gaps with three main contributions:
- Mickey Probe: Instead of predicting a single word, this task asks a model to rank a set of full sentences (assertions) based on their plausibility. It uses Pseudo-Log-Likelihood (PLL) to score sentences, which allows it to evaluate masked language models on whole-sentence semantics rather than just local word patterns.
- Mickey Corpus: To power the probe, they created a massive dataset of 561k sentences across 11 languages. They started with English commonsense facts, used "BERT-attacks" to create plausible-sounding but incorrect distractors, and then used bidirectional translation with a "BT-Cosine" filter to ensure high quality.
- Multilingual Contrastive Pre-training (MCP): This is a new "pretext task" designed to improve sentence-level representations. During training, the model is given a set of assertions in different languages and must identify the one that is actually true. This forces the model to align commonsense concepts across its entire multilingual embedding space.
Datasets and Benchmarks
The research utilizes several key datasets:
- Mickey Corpus: 561k sentences in 11 languages (EN, DE, IT, ES, FR, NL, RU, BG, VI, ZH, HI) used for probing and MCP.
- X-CSQA & X-CODAH: New multilingual versions of the popular CommonsenseQA and CODAH benchmarks. These were created by translating the English originals into 15 other languages, including low-resource languages like Swahili (sw) and Urdu (ur).
Key Insights
- Model "Personalities": Different multilingual models have distinct language preferences. For example, while XLM-R generally performs best, mBERT and d-mBERT actually show stronger performance in French than in German or Dutch, despite having similar training objectives.
- Data Size vs. Quality: Interestingly, the size of the pre-training corpus for a specific language doesn't always correlate with commonsense performance. Russian (ru) has a massive corpus in the CC-100 dataset, yet models often performed worse in Russian than in languages with much smaller training sets, like German. This suggests that vocabulary design and text quality are more critical than raw volume.
- The Power of Contrast: The MCP method proved that "common sense" is largely language-invariant. By training the model to distinguish truth from nonsense across a mix of languages, the authors found that the model's performance improved even on languages it didn't see during the MCP phase.
Results and Performance
- Performance Boost: Applying MCP to the state-of-the-art XLM-R model yielded an average accuracy gain of +2.7% on X-CSQA and +2.3% on X-CODAH across all tested languages.
- Efficiency: Models enhanced with MCP were much more "sample efficient," reaching higher accuracy levels faster during fine-tuning than vanilla models.
- Zero-Shot Transfer: While English-only RoBERTa still leads on English tests, the gap is closing. For X-CSQA, MCP-enhanced XLM-R reached 69.5% accuracy, nearing RoBERTa’s 70.4%.
Limitations and Future Work
The authors acknowledge the issue of cultural bias. Common sense isn't always universal; for instance, the "usual" time for a wedding varies by culture. While they manually filtered out highly controversial or culturally specific examples, they admit that a truly "global" common sense is difficult to define. Future work could involve moving beyond automatic translation to human-authored multilingual datasets and exploring how these improved representations affect downstream tasks like dialogue systems.
Summary updated