Source Language Selection for Cross-Lingual Transfer

310 concepts210 papers1 proposal

When building natural language processing (NLP) systems for low-resource languages, the standard recipe is straightforward: fine-tune a massively multilingual model (like mBERT or XLM-R) on a high-resource "source" language where labeled data is plentiful, and evaluate its zero-shot performance on the target language. Historically, English has been the default source language due to the sheer abundance of high-quality datasets.

However, this English-centric paradigm introduces a significant bottleneck known as English-centric transfer bias. Defaulting to English assumes that data volume trumps all other linguistic factors. In practice, this assumption frequently fails, prompting a shift toward systematic source language selection—the practice of mathematically or empirically choosing the optimal "donor" language to maximize knowledge transfer to a target language.

Why English Fails as a Universal Donor

To understand why source language selection matters, consider the structural distance between languages. Structurally, English is an outlier compared to many of the world's languages: it features minimal morphology, rigid word order, and a specific syntactic profile. If a target language relies heavily on complex agglutinative morphology (like Inuktitut) or flexible word order (like Tamil), transferring task knowledge from English forces the model to overcome a massive typological chasm.

Muller et al. systematically evaluated this phenomenon across 90 language pairs in Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer. Their findings demonstrate that syntactic, morphological, and phonological similarities are far more reliable predictors of successful task transfer than simple vocabulary overlap or raw data size. When models default to English, they sacrifice performance because they ignore these deep structural connections. Similarly, Lauscher et al. showed in From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers that zero-shot transfer from English degrades severely when target languages are typologically distant or have small pretraining corpora.

The core challenge of source language selection is a "cold start" problem: brute-force experimentation—fine-tuning a model on every possible source language to see which one performs best on the target—is computationally prohibitive. Researchers have developed several distinct strategies to predict the best source language efficiently.

Predicting the Optimal Donor Language

Feature-Based Ranking

Instead of guessing or relying on crude heuristics (like geographic proximity), we can treat source selection as a predictive modeling problem. In Choosing Transfer Languages for Cross-Lingual Learning, the authors framed source language selection as a learning-to-rank problem. By training gradient-boosted decision trees on a mixture of linguistic features (mined from typological databases) and dataset statistics, they demonstrated that data-driven ranking consistently outperforms human intuition, identifying high-performing donor languages that human experts might overlook.

Probing Internal Representations

Rather than relying on external typological databases, another approach reads the structural alignment directly from the model's internal representations. Müller-Eberstein et al. introduced DEPPROBE in Probing for Labeled Dependency Trees. DEPPROBE is a lightweight linear probe that extracts fully directed and labeled dependency trees from frozen language model embeddings. By analyzing how easily a model structures grammar across different languages, this method predicts the optimal source language for cross-lingual transfer with 94% accuracy, bypassing the need for full-scale model fine-tuning.

[Raw Embeddings] ──> [DEPPROBE (Lightweight Linear Probe)] ──> [Syntactic Tree Alignment] ──> [Optimal Source Predicted]

Matrix Factorization

When historical experimental data is sparse, we can infer missing transfer performance values using collaborative filtering techniques. Schram et al. addressed this in Performance Prediction via Bayesian Matrix Factorisation for Multilingual Natural Language Processing Tasks. By treating the performance of language pairs as a sparse matrix and injecting linguistic context, their framework predicts transfer success across unobserved language pairs while providing crucial uncertainty estimates that prevent researchers from wasting budget on high-risk experiments.

Acoustic Domain Selection

The need for source selection is not limited to text. In automatic speech recognition (ASR) for low-resource languages, San et al. developed Acoustic Token Distribution Similarity (ATDS) in Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens. ATDS leverages self-supervised acoustic units to accurately predict which untranscribed speech from a donor language will provide the best performance boost, proving that acoustic alignment can be quantified and optimized before training begins.

Multi-Source Selection and Filtering Negative Interference

Often, a single source language does not provide enough coverage. However, simply pooling data from every available language introduces a new failure mode: negative interference. When a model is trained on a wide variety of linguistically diverse sources, conflicting syntactic or morphological patterns can clash, degrading performance on the target language.

To solve this, researchers use iterative selection and adaptive weighting:

By treating the choice of source language as an optimization variable rather than a static default, these frameworks mitigate the curse of multilinguality and provide a data-efficient roadmap for global language coverage.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 229 papers
  2. Version 1 · 2026-10-04 · 183 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_source_language_bias_c441f6ec,
  title = {source language bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/source-language-bias/rdr_c441f6ec8105435eabd0a08c928ad046},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_source_language_bias_c441f6ec_v2,
  title = {source language bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/source-language-bias/rdr_c441f6ec8105435eabd0a08c928ad046/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake