You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models

Tomasz Limisiewicz, Dan Malkin, Gabriel Stanovsky

Abstract

Multilingual models have been widely used for the cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their under-representation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacher-student knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers’ knowledge into a single multilingual student. Our method outperforms standard training methods in low-resource languages and retains performance on high-resource languages while using the same amount of data. If applied widely, our approach can increase the representation of low-resource languages in NLP systems.

Introduction

In the world of Natural Language Processing (NLP), multilingual models like mBERT or XLM-R are the workhorses for global applications. However, they suffer from a significant "representation gap." Because training data is usually scraped from sources like Wikipedia, languages with massive amounts of data (like English or German) dominate the model's attention during pretraining.

Low-resource languages (like Telugu or Basque) are often under-represented, leading to poor performance. A naive fix—simply "balancing" the data by downsampling high-resource languages to match low-resource ones—often backfires. The model loses the rich linguistic information it would have gained from the full high-resource corpus. This creates a challenging trade-off between high- and low-resource language performance.

Methodology: Teacher-Student Distillation

To address this imbalance, we propose a Teacher-Student Distillation framework that leverages "soft labels" to compress information from large datasets into a balanced training regime.

Comparison of standard training using hard-label loss on unbalanced data versus the proposed method using soft-label loss from optimized teachers on balanced data.

The process involves four key steps:

  1. Monolingual Teachers: Individual monolingual models are first trained for each target language using all available data for that language.
  2. Data Balancing: For the multilingual student model, training data for all languages is truncated to match the size of the smallest language's corpus (e.g., 10 million characters).
  3. Soft Labeling (Distillation): Instead of training the student to predict only the "correct" word (hard labels), it is trained to mimic the entire probability distribution of the respective monolingual teacher (soft labels).
  4. Shared Vocabulary: All models share a single multilingual vocabulary, ensuring a consistent probability space for both teacher and student.

This approach allows the student to learn from a small, balanced dataset while "inheriting" the deep linguistic knowledge the teachers gained from their much larger original datasets.

Experimental Setup

We evaluated our method using sub-sampled Wikipedia text across 15 languages, categorized by resource level and script:

  • High-resource (50M-100M chars): English, German, Russian, Spanish, Korean.
  • Low-resource (10M-30M chars): Hungarian, Greek, Vietnamese, Hindi, Turkish, Telugu, Basque, Urdu.

Performance was measured via Mean Reciprocal Rank (MRR) for Masked Language Modeling (MLM), and downstream tasks including Part-of-Speech (POS) tagging and Named Entity Recognition (NER) using the XTREME benchmark.

Key Results and Insights

Our distillation method consistently outperformed standard training, particularly for the lowest-resource languages.

Performance comparison (MRR) across low- and high-resource languages. The proposed method ("Ours") maintains better stability across resource levels compared to standard hard-label (HL) balancing.

Soft Labels as Information Compression

Soft labels act as a "knowledge-dense" proxy. Even though the student sees fewer high-resource sentences, each sentence provides more information because the teacher conveys the nuance of alternative valid tokens in a given context. This allows the model to "punch above its weight," often outperforming standard models that had access to full, unbalanced datasets.

The Script Factor

We compared "Shared Script" (all Latin) vs. "Diverse Script" (Cyrillic, Devanagari, etc.) groups. While sharing a script helps low-resource languages through lexical overlap, diverse scripts can benefit multilingual modeling by forcing the model to learn distinct, high-quality representations for each language. In the "Diverse Script" category, our method achieved an average MRR difference from monolingual baselines of -4.7, outperforming standard training (-5.0) and naive balanced training (-7.2).

Downstream Performance

In NER tasks, the student model showed superior cross-lingual transfer capabilities. For example, in the "Diverse" set, the student achieved a Macro F1 of 33.0, compared to 28.7 for the standard unbalanced model.

Conclusion and Future Work

Our research demonstrates that balancing data doesn't have to mean losing quality if distillation is used. While training individual teachers is computationally expensive, future work could explore re-using existing publicly available monolingual models. Additionally, testing these gains at the scale of massive models like GPT-4 or Llama-3 remains an important next step. By adopting these techniques, we can move toward more equitable and efficient multilingual NLP systems.

Summary updated

Abstract

Multilingual models have been widely used for the cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their under-representation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacher-student knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers’ knowledge into a single multilingual student. Our method outperforms standard training methods in low-resource languages and retains performance on high-resource languages while using the same amount of data. If applied widely, our approach can increase the representation of low-resource languages in NLP systems.
BibTeX
@misc{limisiewicz2022you,
  author = {Tomasz Limisiewicz and Dan Malkin and Gabriel Stanovsky},
  title = {{You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models}},
  year = {2022},
  eprint = {2210.07135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.07135},
}
Figure