Multilingual Lottery Tickets to Pretrain Language Models

Jaeseong Lee, seung-won hwangEMNLP 2023 · OpenReview

Introduction

The "curse of multilinguality" in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages when model capacity is limited. As more languages are added to a fixed-size model like mBERT, performance on individual languages often degrades because they compete for the same parameters. While increasing model capacity can mitigate this interference, it typically leads to significantly higher training and inference costs.

In this work, we propose a novel approach to overcome this curse by pursuing two competing goals: reducing negative interference while maintaining a constant per-language capacity. We introduce Multilingual Lottery Tickets, which leverage the Lottery Ticket Hypothesis and zero-shot neural architecture search (NAS) to find language-specific subnetworks within a larger, scaled model.

Our approach follows a "Scale-then-Search" strategy designed to redistribute parameters rather than simply adding them.

  1. Scaling for Interference Reduction: We first scale the total model capacity (e.g., increasing the number of layers from 12 to 16). This larger "parent" model provides more "room," naturally reducing gradient conflicts between languages during pretraining.
  2. Finding Winning Tickets: Instead of using the entire scaled model for every language, we search for a "winning ticket"—a subnetwork—optimized for each specific language. According to the Lottery Ticket Hypothesis, these subnetworks can match the performance of the full model while being significantly smaller.
  3. Zero-Shot NAS with SNIP: To avoid the prohibitive cost of searching for tickets through repeated retraining, we use SNIP (Single-shot Network Pruning). SNIP identifies critical connections for a specific language using a single gradient pass on a small data sample. This search process is extremely efficient, taking only a few minutes on a single GPU.
Conceptual comparison of (a) standard mBERT, (b) naive scaling, and (c) our proposed approach which keeps per-language capacity constant by finding subnetworks.

Key Insights

  • Parameter Redistribution: By scaling the model and then extracting language-specific tickets, parameters are disseminated across the larger structure. This reduces interference while keeping the inference cost per language comparable to or lower than the original baseline.
  • Linguistic Similarity: Qualitative analysis using UMAP visualizations shows that the discovered tickets preserve linguistic similarity. Languages from the same family (e.g., Romance or Germanic) tend to select similar subnetworks, indicating that the model captures inherent linguistic relationships.
  • Interference Metric: We introduced a cumulative metric ($uc$) to measure language interference more stably than traditional gradient cosine similarity, which tends to be noisy during training.

Experimental Results

We evaluated our method on the XTREME benchmark across 42 languages and 14 language families, focusing on Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.

  • Performance Gains: On NER, our 16-layer multilingual tickets achieved an average F1 score of 84.91, outperforming the 12-layer mBERT baseline (84.21).
  • Efficiency: Despite being derived from a larger model, the inference cost for each language's ticket was 338M FLOPs, lower than the 341.7M FLOPs of the standard mBERT.
  • Search Efficiency: The ticket search cost is negligible, requiring only a few minutes of computation per language.

Conclusion

Multilingual Lottery Tickets offer a scalable solution to the curse of multilinguality. By combining model scaling with efficient subnetwork search, we can reduce negative interference and improve performance across dozens of languages without increasing the per-language computational burden. This approach demonstrates that intelligent parameter redistribution is a powerful alternative to simple model expansion.

Summary updated

Abstract

The curse of multilinguality in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages, especially when the capacity is limited. While increasing the capacity may appear intuitive for overcoming this curse, it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping capacity per each language more or less the same. Specifically, we first scale the model to reduce interference, then search for a per-language subnetwork, or a lottery ticket, with comparable performance to the full model. According to lottery ticket hypothesis, this scale-then-find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters to keep the parameters reduced. Finally, to avoid the cost of multiple retraining for searching multilingual tickets, we explore zero-shot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the inference cost of models for each languages, while boosting the performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is publicly available.
BibTeX
@inproceedings{lee2023multilingual,
  author = {Jaeseong Lee and seung-won hwang},
  title = {{Multilingual Lottery Tickets to Pretrain Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=QG4BWnsX6m},
}
Figure