Abstract
This paper explores multilingual commonsense reasoning through the lens of attention heads in pretrained language models. By introducing XWINO, a standardized multilingual Winograd Schema benchmark, the authors demonstrate that a simple linear classifier trained on attention weights can achieve competitive performance. Crucially, they identify a small, language-agnostic subset of "universal reasoning heads" that drive cross-lingual transfer, suggesting that reasoning capabilities are highly localized within these models.
Introduction
Commonsense reasoning remains a notoriously difficult task in Natural Language Processing, often evaluated using the Winograd Schema Challenge (WSC). In this task, a model must resolve an ambiguous pronoun based on context. While large language models (LLMs) have made significant strides in English, progress in other languages is bottlenecked by a severe lack of labeled data. Furthermore, standard approaches involving full-model fine-tuning on small WSC datasets frequently lead to severe overfitting. This paper addresses these challenges by proposing a robust baseline method that avoids full-model fine-tuning and introduces a unified multilingual benchmark.
The XWINO Benchmark
To evaluate cross-lingual transfer, the authors introduce XWINO, a newly compiled multilingual commonsense reasoning benchmark covering English, French, Japanese, Russian, Chinese, and Portuguese. XWINO standardizes several existing monolingual WSC datasets into a unified binary-choice format where the correct answer is always a direct substring of the input sentence. This strict standardization required filtering out approximately 29% of the original data, resulting in a final dataset of 3,961 examples.
Methodology
Instead of fine-tuning the entire network, the authors propose a lightweight supervised method that treats the internal attention mechanisms of Transformers as a rich feature space:
- Masking: The ambiguous pronoun in a WSC sentence is masked.
- Attention Extraction: The sentence is fed into a frozen pretrained multilingual masked language model (e.g., mBERT or XLM-R). Attention weights between the masked pronoun and the tokens of the two candidate answers are extracted across all layers and heads.
- Feature Engineering: These weights are aggregated (using mean or max pooling) to create fixed-size feature vectors for both candidates.
- Classification: A standard logistic regression classifier is trained on these vectors to predict the correct answer.
Key Insights: Universal Reasoning Heads
The most significant discovery is the existence of Universal Reasoning Heads. The authors found that a very small subset of attention heads specializes in commonsense reasoning across all evaluated languages. These heads are generally located in the higher layers of the model, which are known to encode semantic features.
Interestingly, restricting the linear classifier to use only the top 5 to 16 most important heads actually improves cross-lingual generalization. This "less is more" approach prevents the model from overfitting to language-specific noise found in other heads.
Key Results
- Competitive Performance: Despite its simplicity, the logistic regression over attention weights outperforms several unsupervised baselines and performs competitively with state-of-the-art supervised fine-tuning methods.
- Strong Zero-Shot Transfer: When trained purely on English data and evaluated zero-shot on other languages, the method achieved an average accuracy of 66.7% using XLM-R Large.
- Extreme Pruning: Using only the top 5 common heads out of XLM-R's 384 total heads (just 1.3% of the model's capacity) yielded an average zero-shot accuracy of 66.3%, proving that reasoning capabilities are highly localized.
- Optimal Capacity: The authors identified a "sweet spot" for generalization: using approximately 16 heads yields the best zero-shot cross-lingual performance. Adding more heads increases training accuracy but actively harms test accuracy and cross-lingual transfer.
Limitations and Future Work
The strict filtering pipeline left XWINO somewhat unbalanced; for example, only 16 Chinese examples survived formatting constraints. Additionally, while effective for word-level pronoun resolution, applying this attention-based method to sentence-level tasks (like XCOPA) remains an open question. The authors advocate for future unsupervised methods to focus on head selection and for the community to develop larger, more linguistically diverse WSC datasets.
Summary updated