Abstract
Multilingual Transformers like mBERT and XLM-R demonstrate remarkable zero-shot cross-lingual transfer capabilities. However, accurately quantifying this "transferability" is difficult because standard metrics often conflate a model's task-learning proficiency with its actual cross-lingual mapping skills. We present IGAP (Interlingual transfer GAP), a novel metric designed to isolate the interlingual gap. By decomposing total transfer error, IGAP provides a consistent measure of structural transferability and allows for estimation even in the absence of task-specific data.
Introduction: The Noise in Cross-Lingual Metrics
Multilingual Transformers possess a unique "superpower": zero-shot cross-lingual transfer. A model trained on English data for a task like sentiment analysis can often perform the same task in French or Hindi without seeing a single training example in those languages. However, the standard metric for this—the cross-lingual transfer gap—is often "noisy." It fails to distinguish between a model's failure to learn the underlying task and its failure to transfer that knowledge across languages. As seen in traditional measurements, transfer gaps can be erratic and even negative, complicating model comparison and selection.
Methodology: Decomposing Transfer Error
To address these limitations, we propose decomposing the total cross-lingual transfer error into three distinct components:
- Training Error ($E_{train}$): How well the model learned the training data in the source language.
- Intralingual Generalization Gap ($G_{intra}$): The difference between training and test performance within the same language.
- Interlingual Transfer Gap ($G_{inter}$): The specific loss incurred when moving from the source language to the target language.
IGAP (Interlingual transfer GAP) is defined as the minimum $G_{inter}$ achievable at a specific level of training error. A key innovation is the zero-data estimation of transferability. By using parallel corpora with random labels, we demonstrate that cross-lingual transfer is a structural property of the model's representation space rather than just semantic understanding.
Key Results: Consistency and Model Ranking
Experimental evaluations on benchmarks like XNLI and PAWS-X show that IGAP provides significantly smoother and more consistent transferability curves than traditional metrics.
IGAP successfully ranks models by their transfer capabilities: XLM-R Large consistently exhibits the lowest IGAP (highest transferability), followed by InfoXLM and XLM-R Base, with mBERT trailing. Furthermore, in Transfer Direction Ranking tasks (predicting which target language will perform best), IGAP outperformed standard representation-based baselines like Cosine Similarity, achieving an average accuracy of 75.5% across 15 languages.
Insights and Discussion
Our findings suggest that cross-lingual transfer is fundamentally structural. The ability to transfer random labels proves that the model's internal mapping between languages is the primary driver of success. Interestingly, better representation alignment does not always equate to better transfer, indicating that transferability involves more complex logic than simple spatial proximity. Additionally, advanced fine-tuning algorithms like XTUNE appear to succeed specifically by reducing the IGAP, making the model's internal "bridge" between languages more efficient.
Limitations and Future Work
While IGAP offers a more precise measurement, it currently relies on high-quality parallel data or translations. Future research will investigate how the quality of these translations impacts the metric's reliability and extend the application of IGAP to more complex tasks such as text generation and summarization.
Summary updated