TL;DR
While multilingual language models achieve impressive results on cross-lingual tasks, they do not actually look at the same concepts across different languages. By introducing the Cross-lingual Consistency of Token Attributions (CCTA) framework, this paper reveals that models assign vastly different importance weights to translated synonyms. Crucially, the authors find that higher attribution consistency strongly correlates with better downstream task performance, and that training on Spanish yields more consistent models than training on English.
What Problem Does This Paper Address?
Multilingual Pre-trained Language Models (PLMs) like mBERT and XLM-R are incredibly effective at zero-shot cross-lingual transfer—learning a task in a high-resource language (like English) and applying it to a low-resource language (like Hindi) without extra training. However, why they work so well remains a black box.
Previous interpretability research has focused on whether models represent different languages similarly in their embedding spaces (isomorphism). This paper tackles a more functional question: Do these models value the same concepts across languages when making a prediction? For instance, if a model relies heavily on the word "not" to detect contradiction in an English sentence, does it place the same importance on the translated negative token in the target language? Understanding this "attribution consistency" helps determine if models are truly learning language-agnostic reasoning or just exploiting language-specific shortcuts.
Methods
The authors propose the Cross-lingual Consistency of Token Attributions (CCTA) evaluation framework, which operates in three distinct phases:
- Token Attribution Quantification: To measure how much each token contributes to a model's prediction, the authors use Layer-based Integrated Gradients (LIG). This technique traces gradients from the model's output back to its embedding layer, aggregating the attributions along the embedding dimensions to assign a single importance weight to each token.
- Multilingual Embedding Alignment: Because contextual embeddings reside in language-specific spaces, the authors project token embeddings into a shared, context-free semantic space to bridge the semantic gap, allowing direct cosine-similarity comparisons between tokens of different languages.
- Consistency Measurement via Earth Mover's Similarity: Finally, the framework frames the comparison between a source sentence and its translation as an optimization problem. Using Earth Mover's Similarity (EMS)—a variation of the Wasserstein distance—the framework calculates the maximum possible similarity alignment between the token importance weights of the two parallel sentences.
Datasets
The CCTA framework was evaluated using three state-of-the-art multilingual PLMs (mBERT, XLM, and XLM-R) across three diverse downstream benchmark datasets that feature parallel text:
- XNLI: Cross-lingual Natural Language Inference (evaluated across 13 languages).
- PAWS-X: Cross-lingual Paraphrase Identification (evaluated across 6 languages).
- XQuAD: Cross-lingual Question Answering (evaluated across 11 languages).
Key Insights and Ideas
- Synonyms are Not Valued Equally: Even though multilingual models successfully map translated words close together in embedding space, they treat them very differently during inference. The models frequently rely on entirely different tokens to make the same prediction in different languages.
- The "Spanish Advantage": Surprisingly, English is not the optimal source language for cross-lingual consistency. When models are fine-tuned on Spanish, they achieve significantly more consistent token attributions across target languages—even for linguistically distant target languages like Korean.
- Consistency Drives Performance: There is a strong, positive correlation between attribution consistency and task accuracy. If a model naturally maintains consistent token importance across languages, it performs better on zero-shot transfer. Furthermore, this correlation is stronger on simpler datasets.
Key Results
- Low Baseline Consistency: Across all tasks and models, the cross-lingual token consistency scores were generally low (mostly below 0.5, where 1.0 represents perfect consistency). For instance, when transferring from English, the best target language consistency scores hovered around 0.35 for XNLI, 0.46 for PAWS-X, and 0.36 for XQuAD.
- Strong Performance Correlations: As shown in Figure 2, the correlation coefficients between a model's task performance and its consistency scores reached as high as ~0.8 (e.g., mBERT and XLM-R-Base on the PAWS-X dataset).
- Source Language Impact: When evaluating different source languages on the PAWS-X dataset using XLM-R-Base, Spanish achieved the highest overall consistency score of 0.557, outperforming English (0.479) and German (0.519).
Limitations and Future Work
The authors acknowledge a few limitations and open avenues for future research:
- Requirement for Parallel Corpora: The CCTA framework relies strictly on parallel texts (direct translations) to compute consistency, which limits its application to datasets where exact translations are available.
- Context-Free Alignment: The token alignment step relies on mapping embeddings to a context-free semantic space, which might discard some complex contextual nuances captured by the transformers.
- Unexplained Spanish Superiority: While the empirical data clearly shows Spanish acts as a superior source language for attribution consistency, the underlying linguistic or architectural reasons why remain an open question for future study.
- Enforcing Consistency: A natural next step is to design regularization techniques that explicitly force models to maintain token attribution consistency during training to see if it directly boosts zero-shot performance.
Summary updated