Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?

Junxiang Wang, Xuchao Zhang, Bo Zong, Yanchi Liu, Wei Cheng, Jingchao Ni, Haifeng Chen, Liang Zhao

Abstract

Multilingual Pre-trained Language Models (PLMs) like mBERT and XLM-R have enabled impressive zero-shot cross-lingual transfer. However, the internal mechanisms driving this success remain opaque. This study introduces the Cross-lingual Consistency of Token Attributions (CCTA) framework to investigate whether these models assign similar importance to synonymous words across different languages. Our analysis reveals significant inconsistencies in token attribution, yet demonstrates that higher consistency is a strong predictor of superior downstream task performance.

Introduction

The rise of multilingual PLMs has revolutionized natural language processing by allowing models trained on one language (typically English) to perform tasks in others without additional fine-tuning. While previous research has focused on the alignment of word embeddings, this paper shifts the focus to attribution: identifying which specific tokens most influence a model's final prediction.

A critical question arises: if a model classifies a sentence as "positive" based on the word "excellent" in English, does it rely on the equivalent "excelente" in Spanish? If not, the model may be exploiting language-specific artifacts rather than universal semantic concepts, potentially limiting its robustness and generalizability.

The CCTA Framework

To quantify this phenomenon, the authors developed the CCTA framework, a three-step process designed to measure attribution consistency across languages.

Overview of the CCTA framework for measuring cross-lingual attribution consistency.
  1. Token Attribution Quantification: The framework utilizes Layer-based Integrated Gradients (LIG). LIG accounts for the non-linearities in deep networks by integrating gradients along a path from a baseline (e.g., padding tokens) to the actual input, pinpointing the tokens that truly drive the prediction.
  2. Multi-Lingual Alignment: To compare tokens across different embedding spaces, the authors align embeddings into a shared, context-free semantic space. This allows for mathematical comparison between synonymous tokens in different languages.
  3. Consistency Measurement: The "importance maps" of two sentences are compared using Earth Mover’s Similarity (based on Earth Mover’s Distance). This metric treats attribution as "mass" and calculates the minimum cost to transform the attribution distribution of a source sentence into that of a target translation.

Experimental Setup

The framework was evaluated across three major downstream tasks involving a diverse set of languages:

  • XNLI: Natural Language Inference (13 languages).
  • PAWS-X: Paraphrase Identification (6 languages).
  • XQuAD: Question Answering (11 languages).

Results and Analysis

The study yielded several key insights into the behavior of multilingual models:

The Inconsistency Gap

Multilingual models exhibit a remarkable lack of consistency. As shown in the XNLI results, most consistency scores fall below 0.5, indicating that models often focus on different sentence components depending on the language, even when the underlying meaning is identical.

Consistency scores for different models across various languages in the XNLI task.

Consistency as a Performance Predictor

A pivotal finding is the strong positive correlation (often > 0.7) between attribution consistency and model accuracy. This suggests that for a model to be truly effective in zero-shot settings, it must learn to recognize the importance of tokens in a language-agnostic manner.

Language-Specific Dynamics

The choice of source language significantly impacts consistency. Interestingly, training on Spanish resulted in more consistent attributions across other languages compared to training on English. This suggests that certain languages might be more effective "hubs" for cross-lingual transfer by encouraging the learning of more universal features.

Conclusion and Future Directions

While the CCTA framework provides a novel lens for interpreting multilingual models, it faces limitations such as a dependency on parallel corpora and the use of context-free embeddings for alignment. Future research could explore the use of contextualized embeddings and investigate whether explicitly training models to maximize attribution consistency can directly improve cross-lingual performance. Ultimately, understanding and improving this consistency is a crucial step toward building more robust and truly universal language models.

Summary updated

BibTeX
@misc{wang2021do,
  author = {Junxiang Wang and Xuchao Zhang and Bo Zong and Yanchi Liu and Wei Cheng and Jingchao Ni and Haifeng Chen and Liang Zhao},
  title = {{Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?}},
  year = {2021},
  eprint = {2112.12356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.12356},
}
Figure