Abstract
Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple languages, multilingual models ideally align representations, allowing what is learned in one language to generalise to others. Prior research has emphasised the importance of parallel data and shared vocabulary elements as key factors for such alignment. In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages. Furthermore, we find that this trend is amplified with scale: with large enough models or long enough training, we observe that bilingual training data with a 90/10 language split yields better performance on both languages than a balanced 50/50 split. Building on these insights, we design training schemes that can improve performance in all cloned languages, even without altering the training data. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
Introduction
Multilingual models often face the "curse of multilinguality," where increasing the number of supported languages leads to a performance degradation on individual languages due to competition for model capacity. Conventional wisdom suggests that balanced datasets are the solution to ensure fair representation. However, this research explores whether the ratio of data between languages—specifically language imbalance—can actually serve as a driver for cross-lingual generalisation.
To isolate the effects of data distribution from linguistic complexities, the researchers utilized a "Cloned Languages" framework. By creating a second version of English with a disjoint vocabulary but identical syntax and semantics, they could attribute performance differences solely to training dynamics. The primary metric used is Token Efficiency (TEff), which quantifies how many multilingual tokens a model's performance is worth compared to a monolingual baseline.
The Imbalance Paradox in Cloned Languages
The core finding of the study is that language imbalance can trigger superior cross-lingual generalisation. In a balanced 50/50 split, models often fail to align representations, treating the cloned languages as separate entities. Conversely, in a 90/10 split, the model is incentivized to reuse the internal circuits developed for the dominant language to process the minority language.
As shown in Figure 2, this effect is amplified by scale. As models grow in size or are trained for longer durations, the TEff for the minority language in a 90/10 split significantly exceeds that of a balanced split, often reaching values over 2.0. This indicates that the rare language benefits massively from the dominant language's data, effectively "piggybacking" on its learned representations.
Real-World Generalisation: English and French
Moving beyond synthetic clones, the researchers tested these dynamics on real-world English and French pairs. While the effect was less dramatic than with clones—likely due to linguistic "friction" such as differing word orders—imbalance still provided a boost to the low-resource language.
Figure 5 illustrates that even in real languages, a lower probability of the target language ($p(FR)$) can lead to improved Token Efficiency, suggesting that the model leverages the high-resource language (English) to better represent the low-resource one. However, the "Imbalance Paradox" where the dominant language also improves is less clear in real-world settings compared to the perfectly symmetric cloned environment.
Discussion and Conclusion
The study reveals that language imbalance is not merely a challenge to be overcome but can be a powerful tool for representation alignment. By forcing models to share internal representations, imbalance facilitates the transfer of knowledge from high-resource to low-resource languages.
Future work is needed to determine if these dynamics hold at the scale of frontier LLMs and across more linguistically diverse pairs, such as English and Chinese. Nevertheless, these insights suggest that strategic data scheduling—starting with an imbalanced "warm-up" before moving to a balanced diet—could be a viable path toward more efficient multilingual training.
Paper Citation: Schäfer, A., Ravfogel, S., Hofmann, T., Pimentel, T., & Schlag, I. (2024). The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments. arXiv:2404.07982v4.
Summary updated