It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

Alexey Tikhonov, Max Ryabinin

TL;DR

Instead of finetuning massive language models for commonsense reasoning—which often leads to severe overfitting on small datasets—this paper shows that a simple logistic regression trained on raw self-attention weights can perform robust cross-lingual pronoun resolution. Furthermore, the authors discover that cross-lingual commonsense reasoning is concentrated in a tiny, universal subset of attention heads (just ~1.3% of the model) located in the upper layers of multilingual transformers.

What Problem Does This Paper Address?

The Winograd Schema Challenge (WSC) tests whether models possess commonsense reasoning. A classic example: "The town councilors refused to give the demonstrators a permit because they feared violence." Resolving whether "they" refers to the councilors or the demonstrators requires real-world commonsense knowledge rather than superficial statistical correlations.

While large masked language models (MLMs) can solve WSC, existing approaches typically require full finetuning on large English corpora. When applied to smaller non-English datasets, standard finetuning quickly memorizes the training data, leading to abysmal out-of-sample and zero-shot cross-lingual transfer. Additionally, non-English WSC datasets were previously fragmented across disparate formats and conflicting task formulations.

Methods

The authors propose a lightweight, feature-based supervised approach:

  1. Masking & Extraction: Given a sentence with a target pronoun and two candidate antecedents, mask the pronoun and run the sentence through a frozen multilingual encoder (multilingual BERT or XLM-R-Large).
  2. Attention Representation: For every layer $L$ and attention head $H$, extract the attention weights between the pronoun token and the candidate answer tokens.
  3. Pooling & Vector Formulation:
  4. Use mean-pooling across candidate tokens to obtain a fixed-size vector per candidate.
  5. Look specifically at attention from candidate tokens to the masked pronoun (as identified in their ablation study).
  6. Subtract the second candidate’s attention vector from the first candidate’s vector (feature difference works significantly better than concatenation).
  7. Classification: Train a standard, off-the-shelf logistic regression classifier on these difference vectors to predict whether Candidate 1 or Candidate 2 is the correct antecedent.

Datasets

The authors assemble and release XWINO, a standardized multilingual WSC benchmark spanning six languages: English, French, Japanese, Russian, Chinese, and Portuguese.

  • They harmonized diverse source datasets (e.g., DPR, original WSC285, RussianSuperGLUE) into a strict, unified binary-choice format where both candidate options and the pronoun appear as exact substrings of the input sentence.
  • After filtering inconsistent annotations (such as questions instead of pronouns or missing substrings), the final benchmark retains 3,961 examples (71% of the raw collected data).

Key Insights and Ideas

  • Universal Reasoning Heads: Across languages, reasoning ability is concentrated in a remarkably small subset of attention heads in the model’s upper layers (layers known to encode semantic features). Using only the top-5 common heads in XLM-R-Large (just 5 out of 384 heads, or ~1.3%) matches or exceeds the performance of using all heads across almost all languages.
  • Unsupervised Methods Suffer from Head Selection: The unsupervised Masked Attention Score (MAS) baseline tallies raw attention across all heads. Restricting MAS to just the top-5 discovered heads boosted its average accuracy from 56.2% to 65.6%, demonstrating that attention heads already encode the reasoning signal—unsupervised methods just get drowned out by noisy, non-specialized heads.
  • Less is More for Cross-Lingual Transfer: Training accuracy increases monotonically as more attention heads are added, but cross-lingual zero-shot transfer peaks around 16 heads. Beyond that, linear models overfit to language-specific idiosyncrasies.

Key Results

  • Resisting Overfitting: On smaller subsets (e.g., French, Japanese, Russian), baseline finetuning (Kocijan et al., 2019) hit 100% training accuracy but collapsed to near-random test/zero-shot performance (often 33%–50%). The attention-head classifier avoided catastrophic overfitting and achieved stable generalization.
  • Encoder Scaling: Moving from mBERT to XLM-R-Large yielded major gains: our method's cross-lingual zero-shot average accuracy improved from ~53% on mBERT to ~67% on XLM-R-Large (e.g., English-trained models scored 67.5% on French, 69.1% on Japanese, 70.4% on Russian, and 66.8% on Portuguese).
  • Competitiveness: Despite relying on a simple linear classifier over frozen representations, the method achieved results competitive with full finetuning while using a fraction of the computational and parameter overhead.

Limitations and Future Work

  • Strictly Token-Level: The method is formulated for word- or phrase-level coreference/pronoun resolution and cannot be applied directly out-of-the-box to sentence-level commonsense benchmarks like XCOPA.
  • Data Scarcity in Some Languages: Some language splits remain very small (e.g., Chinese had only 16 usable examples after filtering), resulting in high experimental variance and poor local classifier training for those subsets.
  • Filtering Discard Rate: The preprocessing pipeline dropped ~29% of collected examples due to formatting mismatches, highlighting the ongoing need for larger, natively constructed multilingual commonsense benchmarks.

Summary updated

BibTeX
@inproceedings{tikhonov2021it,
  author = {Alexey Tikhonov and Max Ryabinin},
  title = {{It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=MnwD_ykoLy5},
}

Loading related content...

Figure