Measuring Cross-Lingual Transferability of Multilingual Transformers

Heyan Huang, Zewen Chi, Xian-Ling MaoIEEE Transactions on Audio Speech and Language Processing 2025 · DOI

TL;DR

This paper introduces IGAP, a novel metric for measuring cross-lingual transferability in multilingual Transformer models. IGAP addresses the limitations of existing methods by considering training error and enabling transferability estimation even without end-task data, offering a more robust way to understand and compare these powerful models.

What Problem Does This Paper Address?

Multilingual Transformers, like BERT and XLM-R, have shown impressive capabilities in transferring knowledge across languages, a phenomenon known as cross-lingual transferability. This is crucial for building NLP systems that work well in diverse linguistic settings, especially for low-resource languages. However, current methods for measuring this transferability are insufficient. They often conflate transferability with overall performance, making it hard to pinpoint what makes a model good at transferring knowledge. This lack of a reliable metric hinders our ability to analyze, compare, and ultimately improve these multilingual models. The authors aim to fill this gap by proposing a new metric that accurately quantifies cross-lingual transferability.

Methods

The core contribution is the IGAP (Interlingual Gap and Training error) metric. IGAP decomposes the cross-lingual transfer error into three components:

  • Interlingual Transfer Gap (Ginter): This measures how much knowledge is lost when transferring from a source to a target language. It's calculated using translated examples to ensure consistent task difficulty.
  • Intralingual Generalization Gap (Gintra): This captures the model's ability to generalize to unseen examples within the target language.
  • Training Error (Etrain): This reflects how well the model learned the task on the source language training data.

IGAP's key innovation is its focus on the interlingual transfer gap while explicitly considering the training error. This allows it to estimate transferability even when end-task data is unavailable by using parallel corpora and randomly generated labels. The metric is designed to be comparable across different models, fine-tuning algorithms, and transfer directions.

Datasets

The paper uses several standard datasets for evaluation:

  • XNLI: A large-scale natural language inference (NLI) dataset with human-translated validation sets in multiple languages. Used for measuring transferability and transfer direction ranking.
  • PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. Used for evaluating transferability.
  • MNLI: Multi-Genre Natural Language Inference dataset. Used for larger-scale fine-tuning experiments.
  • FLORES-101: A parallel corpus used for estimating transferability without end-task data.

Key Insights and Ideas

  • IGAP offers a more nuanced view: Unlike simple performance metrics or the "cross-lingual transfer gap," IGAP disentangles transferability from raw performance, providing a clearer picture of a model's cross-lingual learning ability.
  • Transferability can be estimated without task data: The ability to use parallel corpora and random labels to estimate transferability is a significant advantage, especially for low-resource languages where labeled data is scarce.
  • Better alignment doesn't guarantee better transfer: The paper's experiments reveal that models with more aligned representations across languages don't necessarily exhibit better cross-lingual transferability. This challenges a common assumption in the field.
  • Cross-lingual transfer methods implicitly reduce IGAP: Fine-tuning algorithms and model architectures designed to improve cross-lingual performance appear to do so by effectively reducing the IGAP score.
  • Multilingual Transformers can "memorize" and transfer knowledge: Even with randomly generated labels (simulating novel, unseen knowledge), models can transfer this "knowledge" to other languages, suggesting a powerful memorization and generalization capability.

Key Results

  • IGAP outperforms baselines: IGAP consistently shows clearer and more consistent trends in measuring transferability compared to the cross-lingual transfer gap and representation-based metrics.
  • Transfer Direction Ranking (TDR): IGAP achieved the best TDR accuracy in 13 out of 15 languages, demonstrating its effectiveness in predicting which target language will perform best for a given source language, even without end-task data.
  • Model Comparisons: XLM-Rlarge generally shows lower IGAP scores (indicating better transferability) than XLM-R and InfoXLM, aligning with its higher test accuracy.
  • Fine-tuning Algorithms: Algorithms like GAUSSIAN and XTUNE improve cross-lingual transfer performance and consistently lead to lower IGAP scores compared to vanilla fine-tuning.
  • Memorization Effect: Even with up to 100% randomly generated labels, models achieved significantly better-than-random accuracy and IGAP scores, indicating that they can transfer even this "noisy" knowledge.

Limitations and Future Work

  • Reliance on Parallel Data: IGAP's accuracy depends on the quality of parallel sentences or translated end-task examples. The authors caution that researchers must carefully select these resources to avoid misleading scores.
  • Potential for Misleading Scores: The use of translated data introduces a risk of obtaining inaccurate IGAP scores if the translations are poor or "poisoned."
  • Future Work: The authors suggest extending IGAP to analyze multilingual Transformers on more NLP tasks, such as language generation. They also plan to investigate how the quality of translated examples and parallel sentences specifically impacts their metric.

Summary updated

Abstract

Recent studies have exhibited remarkable cross-lingual abilities of pre-trained multilingual Transformers, especially cross-lingual transferability. However, current methods do not measure cross-lingual transferability well, hindering the understanding of multilingual Transformers. In this paper, we propose <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IGap</small>, a cross-lingual transferability metric for multilingual Transformers. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IGap</small> utilizes parallel data to measure cross-lingual transferability while taking training error into consideration, and can also estimate transferability without end-task data. Experimental results show that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IGap</small> outperforms baseline metrics for transferability measuring and transfer direction ranking. Besides, we conduct extensive systematic experiments where we compare transferability among various multilingual Transformers, fine-tuning algorithms, and transfer directions. More importantly, our results reveal three findings about cross-lingual transfer, which helps us to better understand multilingual Transformers.
BibTeX
@article{huang2025measuring,
  author = {Heyan Huang and Zewen Chi and Xian-Ling Mao},
  title = {{Measuring Cross-Lingual Transferability of Multilingual Transformers}},
  journal = {IEEE Transactions on Audio Speech and Language Processing},
  year = {2025},
  doi = {10.1109/taslpro.2025.3577329},
}
Figure