Abstract
Commonsense reasoning research has so far been limited to English. We aim to evaluate and improve popular multilingual language models (ML-LMs) to help advance commonsense reasoning (CSR) beyond English. We collect the Mickey Corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and improving ML-LMs. We propose Mickey Probe, a language-agnostic probing task for fairly evaluating the common sense of popular ML-LMs across different languages. In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 15 other languages, so that we can evaluate popular ML-LMs for cross-lingual commonsense reasoning. To improve the performance beyond English, we propose a simple yet effective method -- multilingual contrastive pre-training (MCP). It significantly enhances sentence representations, yielding a large performance gain on both benchmarks.
Introduction
While commonsense reasoning (CSR) is a cornerstone of human intelligence, AI research has largely treated it as an English-only problem. Most "common sense" benchmarks—like CommonsenseQA or SWAG—are designed in English, creating a bottleneck for global AI development. We don't actually know if our Multilingual Language Models (ML-LMs) like mBERT or XLM-R truly "understand" the physical world, or if they are just memorizing English-specific patterns.
Existing evaluation methods, such as the LAMA probe, are often limited to single-token predictions (e.g., "Birds have [mask]"). This approach fails in many non-English languages where concepts are multi-token or where grammar makes single-word masking unfair. Furthermore, standard pre-training for these models focuses on word-level masking, which doesn't necessarily teach the model how to evaluate the logic of an entire sentence.
Methodology
To address these challenges, we introduce two main innovations: Mickey Probe and Multilingual Contrastive Pre-training (MCP).
Mickey Probe: A Language-Agnostic Evaluation
Instead of filling in a blank, the Mickey Probe provides the model with a set of similar-looking assertions (e.g., "The effect of reading the news is learning about the world" vs. "The effect of reading the news is lying about the world"). The model must rank these based on their pseudo-log-likelihood (PLL)—essentially, how "natural" or "plausible" the sentence sounds to the model. This method is language-agnostic and avoids the pitfalls of tokenization-dependent masking.
Multilingual Contrastive Pre-training (MCP)
To improve reasoning, we propose a new pre-training objective. The model is shown a set of $N$ assertions in $N$ different languages. Only one is a "true" commonsense statement; the others are distractors. The model must learn to project these sentences into a shared embedding space and select the correct one. This forces the model to develop better sentence-level representations that transcend specific languages, moving beyond simple word-level masking.
Datasets and Resources
We released several major resources to facilitate multilingual CSR research:
- Mickey Corpus: A massive parallel corpus of 561k sentences across 11 languages used for both probing and training.
- X-CSQA & X-CODAH: Multilingual versions of two gold-standard English benchmarks, translated into 15 additional languages including Chinese, German, Russian, Vietnamese, and low-resource languages like Swahili and Urdu.
- Quality Control: To ensure translation accuracy, we utilized "BT-Cosine"—a method that checks if a sentence translated to another language and back to English still retains its original meaning.
Key Results and Insights
Our evaluation yielded several critical insights:
- XLM-R Dominance: XLM-R emerged as the strongest baseline model for multilingual common sense.
- MCP Effectiveness: Adding the MCP objective to XLM-R led to a +2.7% absolute accuracy gain on the X-CSQA-avg benchmark. Notably, on low-resource languages like Swahili and Urdu, MCP provided a 2–3 percentage point improvement.
- Cross-Lingual Generalization: MCP improved performance even on languages not included in the MCP training set, suggesting the model learns a universal "reasoning" feature.
- Sentence-Level Logic: Sentence-level objectives are superior to word-level masking for understanding scene plausibility.
Conclusion and Future Work
Despite these gains, a significant gap remains between AI and human performance, particularly in categories like "Idioms" and "Polysemy." Furthermore, "common sense" is not always universal; cultural differences can introduce bias. While we filtered for culture-specific examples, defining a truly "culture-invariant" common sense remains a challenge. Future work will investigate handling culture-specific knowledge and expanding these methods to broader Natural Language Understanding (NLU) tasks.
Summary updated