Abstract
Multilingual Large Language Models (LLMs) often rely on cross-lingual transfer (XLT) to perform tasks in low-resource languages. However, selecting the optimal source language for transfer is a significant challenge, as the default choice (English) is frequently suboptimal. We present X-SNS (Cross-Lingual Transfer Prediction through Sub-Network Similarity), a model-oriented method that predicts the best source language by measuring the similarity between language-specific sub-networks within the model. Using Fisher Information to identify critical parameters, X-SNS identifies effective source languages using only small amounts of unlabeled text. Our results across the XTREME benchmark demonstrate that X-SNS consistently outperforms traditional linguistic-based predictors and identifies better source languages than English in the majority of low-resource cases.
Introduction
The proliferation of multilingual models like mBERT and XLM-RoBERTa has enabled task performance across hundreds of languages. Yet, for low-resource languages, the standard practice of fine-tuning on English and transferring to the target language often yields mediocre results. Finding a better source language typically requires exhaustive "brute-force" testing, which is computationally prohibitive. Existing transfer predictors often rely on external linguistic databases (e.g., Lang2Vec) or surface-level lexical statistics, which may not capture how a specific model's internal architecture processes different languages.
Methodology: Sub-Network Similarity
X-SNS shifts the focus from external linguistic features to the model's internal state. The core hypothesis is that effective knowledge transfer occurs between languages that utilize similar "sub-networks"—sets of parameters critical for processing each language.
The X-SNS pipeline consists of five stages:
- Gradient Calculation: Gradients are computed for model parameters using a small sample of raw text (approx. 1,000 sentences).
- Fisher Information Approximation: These gradients estimate the "importance" of each parameter.
- Sub-Network Extraction: A binary mask is created by selecting the top $p\%$ (optimally 15%) of the most important parameters.
- Similarity Measurement: The Jaccard coefficient measures the overlap between the source and target language masks.
- Prediction: This similarity score ranks potential source languages.
This approach is highly data-efficient and model-oriented, requiring no labeled task data for the prediction phase.
Experimental Setup
We evaluated X-SNS using XLM-RoBERTa-base across five diverse tasks from the XTREME benchmark: Named Entity Recognition (NER), Part-of-Speech tagging (POS), Natural Language Inference (NLI), Paraphrase Identification (PI), and Question Answering (QA). The method was also validated on mBERT and mT5 to ensure robustness across different architectures.
Results and Discussion
X-SNS demonstrates a clear advantage over traditional baselines. It achieved a 4.6% average improvement in NDCG@3 across all tasks compared to methods like Lang2Vec and Lexical Divergence.
Key findings include:
- Beyond English: For many languages, particularly those with distinct scripts like Arabic (AR), Greek (EL), or Thai (TH), X-SNS correctly identifies source languages that significantly outperform English. In 11 out of 15 low-resource cases, X-SNS found a better source than English, leading to an average F1 score increase of 1.8 points.
- Model-Specific Insights: The internal parameter importance serves as a more reliable signal for transferability than static linguistic features.
- Domain Sensitivity: Performance improves when sub-networks are extracted using text from the same domain as the target task, highlighting the importance of task-relevant internal activations.
Conclusion
X-SNS provides a robust, data-efficient framework for optimizing cross-lingual transfer. By looking "under the hood" of multilingual models, it identifies transfer opportunities that traditional linguistic methods miss. While future work will address the storage overhead of gradient information and scaling to larger LLMs, X-SNS represents a significant step toward more efficient and effective multilingual NLP.
Summary updated