Introduction
The "curse of multilinguality" in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages when model capacity is limited. As more languages are added to a fixed-size model like mBERT, performance on individual languages often degrades because they compete for the same parameters. While increasing model capacity can mitigate this interference, it typically leads to significantly higher training and inference costs.
In this work, we propose a novel approach to overcome this curse by pursuing two competing goals: reducing negative interference while maintaining a constant per-language capacity. We introduce Multilingual Lottery Tickets, which leverage the Lottery Ticket Hypothesis and zero-shot neural architecture search (NAS) to find language-specific subnetworks within a larger, scaled model.
Methodology: Scale-then-Search
Our approach follows a "Scale-then-Search" strategy designed to redistribute parameters rather than simply adding them.
- Scaling for Interference Reduction: We first scale the total model capacity (e.g., increasing the number of layers from 12 to 16). This larger "parent" model provides more "room," naturally reducing gradient conflicts between languages during pretraining.
- Finding Winning Tickets: Instead of using the entire scaled model for every language, we search for a "winning ticket"—a subnetwork—optimized for each specific language. According to the Lottery Ticket Hypothesis, these subnetworks can match the performance of the full model while being significantly smaller.
- Zero-Shot NAS with SNIP: To avoid the prohibitive cost of searching for tickets through repeated retraining, we use SNIP (Single-shot Network Pruning). SNIP identifies critical connections for a specific language using a single gradient pass on a small data sample. This search process is extremely efficient, taking only a few minutes on a single GPU.
Key Insights
- Parameter Redistribution: By scaling the model and then extracting language-specific tickets, parameters are disseminated across the larger structure. This reduces interference while keeping the inference cost per language comparable to or lower than the original baseline.
- Linguistic Similarity: Qualitative analysis using UMAP visualizations shows that the discovered tickets preserve linguistic similarity. Languages from the same family (e.g., Romance or Germanic) tend to select similar subnetworks, indicating that the model captures inherent linguistic relationships.
- Interference Metric: We introduced a cumulative metric ($uc$) to measure language interference more stably than traditional gradient cosine similarity, which tends to be noisy during training.
Experimental Results
We evaluated our method on the XTREME benchmark across 42 languages and 14 language families, focusing on Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.
- Performance Gains: On NER, our 16-layer multilingual tickets achieved an average F1 score of 84.91, outperforming the 12-layer mBERT baseline (84.21).
- Efficiency: Despite being derived from a larger model, the inference cost for each language's ticket was 338M FLOPs, lower than the 341.7M FLOPs of the standard mBERT.
- Search Efficiency: The ticket search cost is negligible, requiring only a few minutes of computation per language.
Conclusion
Multilingual Lottery Tickets offer a scalable solution to the curse of multilinguality. By combining model scaling with efficient subnetwork search, we can reduce negative interference and improve performance across dozens of languages without increasing the per-language computational burden. This approach demonstrates that intelligent parameter redistribution is a powerful alternative to simple model expansion.
Summary updated