The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments

Anton Schäfer, Shauli Ravfogel, Thomas Hofmann, Tiago Pimentel, Imanol Schlag

Abstract

Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple languages, multilingual models ideally align representations, allowing what is learned in one language to generalise to others. Prior research has emphasised the importance of parallel data and shared vocabulary elements as key factors for such alignment. In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages. Furthermore, we find that this trend is amplified with scale: with large enough models or long enough training, we observe that bilingual training data with a 90/10 language split yields better performance on both languages than a balanced 50/50 split. Building on these insights, we design training schemes that can improve performance in all cloned languages, even without altering the training data. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.

Introduction

Multilingual Large Language Models (LLMs) aim to be language-agnostic, ideally generalizing reasoning capabilities across linguistic boundaries. Traditionally, this cross-lingual generalization has been attributed to parallel data and shared vocabulary "anchor points." However, many state-of-the-art models are trained on highly imbalanced datasets dominated by English, yet they exhibit remarkable performance in low-resource languages. This study investigates whether language imbalance itself—the presence of a dominant majority language—serves as a catalyst for internal representation alignment and improved generalization.

Methodology: The Cloned Language Framework

To isolate the effects of language ratio from linguistic noise (e.g., grammar, morphology), the authors utilized "Cloned Languages." By taking an English dataset (Books3) and replacing every word with unique, meaningless tokens, they created a synthetic twin language with identical syntax and semantics. This controlled environment allows for precise measurement of how training ratios (e.g., 50/50 vs. 90/10) affect performance.

The primary metric used is Token Efficiency (TEff). A TEff of 2.0 indicates that the model learns as much from 100 multilingual tokens as it would from 200 monolingual tokens, signifying a high degree of cross-lingual transfer.

Results: The Power of Imbalance

The experiments reveal a striking "imbalance advantage." In a balanced 50/50 split, models often treat the two languages as separate tasks, leading to negative interference where learning one language hinders the other. Conversely, in a 90/10 split, the model first establishes a robust internal "concept space" using the majority language. The minority language then "latches onto" this existing map, leading to significant efficiency gains.

Token Efficiency (TEff) in cloned experiments across training duration and model size.

As shown in Figure 3, the TEff for the minority language ($EN_2$) in a 90/10 split consistently exceeds 2.0. Furthermore, this effect is amplified by scale: as model size increases from "mini" to "medium," the efficiency of the minority language continues to rise, suggesting that larger models are better at leveraging majority-language knowledge to represent minority-language tokens.

Scaling and Real-World Generalization

The study extended these findings to real-world language pairs, specifically English and French. While linguistic differences introduce complexity, the core trend persists: the minority language benefits from the presence of a frequent majority language.

Perplexity and Token Efficiency for English-French experiments as a function of language ratio.

Figure 4 demonstrates that as the proportion of French ($p(FR)$) decreases, its Token Efficiency increases, mirroring the cloned experiments. Interestingly, the researchers observed a "90/10 Paradox" in long-duration training runs, where an imbalanced split can eventually outperform a balanced split on both languages. This suggests that the majority language provides a stable "anchor" that prevents the model from diverging into language-specific silos.

Conclusion and Future Work

This research challenges the intuition that balanced data is always optimal for multilingual training. Instead, language imbalance can force models to align representations, boosting the performance of low-resource languages. However, limitations remain: the "alignment" effect is less pronounced in real languages than in clones at current scales, and the impact of linguistic distance (e.g., English vs. Chinese) requires further investigation. Future work should explore whether these dynamics hold for models with billions of parameters and how training schedules can be optimized to exploit the imbalance boost.

Summary updated

Abstract

Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple languages, multilingual models ideally align representations, allowing what is learned in one language to generalise to others. Prior research has emphasised the importance of parallel data and shared vocabulary elements as key factors for such alignment. In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages. Furthermore, we find that this trend is amplified with scale: with large enough models or long enough training, we observe that bilingual training data with a 90/10 language split yields better performance on both languages than a balanced 50/50 split. Building on these insights, we design training schemes that can improve performance in all cloned languages, even without altering the training data. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
BibTeX
@misc{schafer2024role,
  author = {Anton Schäfer and Shauli Ravfogel and Thomas Hofmann and Tiago Pimentel and Imanol Schlag},
  title = {{The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments}},
  year = {2024},
  eprint = {2404.07982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.07982},
}
Figure