When building natural language processing (NLP) systems for low-resource languages, the standard recipe is straightforward: fine-tune a massively multilingual model (like mBERT or XLM-R) on a high-resource "source" language where labeled data is plentiful, and evaluate its zero-shot performance on the target language. Historically, English has been the default source language due to the sheer abundance of high-quality datasets.
However, this English-centric paradigm introduces a significant bottleneck known as English-centric transfer bias. Defaulting to English assumes that data volume trumps all other linguistic factors. In practice, this assumption frequently fails, prompting a shift toward systematic source language selection—the practice of mathematically or empirically choosing the optimal "donor" language to maximize knowledge transfer to a target language.
Why English Fails as a Universal Donor
To understand why source language selection matters, consider the structural distance between languages. Structurally, English is an outlier compared to many of the world's languages: it features minimal morphology, rigid word order, and a specific syntactic profile. If a target language relies heavily on complex agglutinative morphology (like Inuktitut) or flexible word order (like Tamil), transferring task knowledge from English forces the model to overcome a massive typological chasm.
Muller et al. systematically evaluated this phenomenon across 90 language pairs in Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer. Their findings demonstrate that syntactic, morphological, and phonological similarities are far more reliable predictors of successful task transfer than simple vocabulary overlap or raw data size. When models default to English, they sacrifice performance because they ignore these deep structural connections. Similarly, Lauscher et al. showed in From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers that zero-shot transfer from English degrades severely when target languages are typologically distant or have small pretraining corpora.
The core challenge of source language selection is a "cold start" problem: brute-force experimentation—fine-tuning a model on every possible source language to see which one performs best on the target—is computationally prohibitive. Researchers have developed several distinct strategies to predict the best source language efficiently.
Predicting the Optimal Donor Language
Feature-Based Ranking
Instead of guessing or relying on crude heuristics (like geographic proximity), we can treat source selection as a predictive modeling problem. In Choosing Transfer Languages for Cross-Lingual Learning, the authors framed source language selection as a learning-to-rank problem. By training gradient-boosted decision trees on a mixture of linguistic features (mined from typological databases) and dataset statistics, they demonstrated that data-driven ranking consistently outperforms human intuition, identifying high-performing donor languages that human experts might overlook.
Probing Internal Representations
Rather than relying on external typological databases, another approach reads the structural alignment directly from the model's internal representations. Müller-Eberstein et al. introduced DEPPROBE in Probing for Labeled Dependency Trees. DEPPROBE is a lightweight linear probe that extracts fully directed and labeled dependency trees from frozen language model embeddings. By analyzing how easily a model structures grammar across different languages, this method predicts the optimal source language for cross-lingual transfer with 94% accuracy, bypassing the need for full-scale model fine-tuning.
[Raw Embeddings] ──> [DEPPROBE (Lightweight Linear Probe)] ──> [Syntactic Tree Alignment] ──> [Optimal Source Predicted]
Matrix Factorization
When historical experimental data is sparse, we can infer missing transfer performance values using collaborative filtering techniques. Schram et al. addressed this in Performance Prediction via Bayesian Matrix Factorisation for Multilingual Natural Language Processing Tasks. By treating the performance of language pairs as a sparse matrix and injecting linguistic context, their framework predicts transfer success across unobserved language pairs while providing crucial uncertainty estimates that prevent researchers from wasting budget on high-risk experiments.
Acoustic Domain Selection
The need for source selection is not limited to text. In automatic speech recognition (ASR) for low-resource languages, San et al. developed Acoustic Token Distribution Similarity (ATDS) in Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens. ATDS leverages self-supervised acoustic units to accurately predict which untranscribed speech from a donor language will provide the best performance boost, proving that acoustic alignment can be quantified and optimized before training begins.
Multi-Source Selection and Filtering Negative Interference
Often, a single source language does not provide enough coverage. However, simply pooling data from every available language introduces a new failure mode: negative interference. When a model is trained on a wide variety of linguistically diverse sources, conflicting syntactic or morphological patterns can clash, degrading performance on the target language.
To solve this, researchers use iterative selection and adaptive weighting:
- Iterative Search Filtering: In NLNDE at SemEval-2024 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis, Schuetze et al. implemented an iterative source language selection strategy using forward and backward search. By systematically adding or removing auxiliary languages based on their empirical performance impact, they filtered out harmful data and maximized positive transfer for low-resource African languages. This strategy is supported by Adelani et al. in SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval), which notes that selective multilingual source integration prevents cross-lingual dilution by focusing strictly on the most relevant linguistic data.
- Adaptive Knowledge Distillation: Rather than hard filtering, we can dynamically weight the influence of different sources throughout the training process. Saleh et al. introduced this concept for low-resource machine translation in Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation. Instead of relying on a single teacher model, their framework distills knowledge from an ensemble of source-language teachers, dynamically adjusting each teacher's weight based on its real-time performance on the target language.
By treating the choice of source language as an optimization variable rather than a static default, these frameworks mitigate the curse of multilinguality and provide a data-efficient roadmap for global language coverage.