Abstract
Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties. In contrast to prior findings, we find that compression may improve model robustness over dense models. We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.
Introduction
As Large Language Models (LLMs) expand to support a global user base, they encounter the "curse of multilinguality": the requirement for exponentially more parameters to maintain performance as more languages are added. To deploy these massive models on edge devices or within resource-constrained environments, researchers typically turn to compression techniques like pruning—the process of removing less significant weights from a network.
Conventionally, model compression is viewed as a necessary trade-off, where a reduction in size inevitably leads to a reduction in accuracy, particularly for "sub-populations" or low-resource languages that have less representation in the training data. This paper challenges that narrative, demonstrating that moderate pruning can serve as a "secret sauce" for multilingual performance, enhancing results for low-resource languages and significantly boosting model robustness against data perturbations.
Experimental Framework
To investigate the relationship between compression and multilinguality, the researchers focused on the task of Named Entity Recognition (NER) using mBERT (multilingual BERT). They employed Iterative Magnitude Pruning (IMP) during the fine-tuning stage. IMP is a cyclical process where a model is trained, the weights with the smallest magnitudes are removed, and the remaining weights are retrained to recover performance.
The study compared two distinct pruning strategies to isolate the impact of different model components:
- Partial Pruning: Pruning only the dense layers while keeping the embedding layers (which map tokens to vectors) fully intact.
- Including Embeddings: Pruning both the dense layers and the embedding layers.
Datasets and Language Selection
The researchers utilized the WikiAnn dataset, a silver-standard NER benchmark. They selected 40 languages from the XTREME benchmark to ensure a diverse representation across three axes:
- Resourcedness: Training sets ranging from 100 to 20,000 sentences.
- Script: Including Latin, Arabic, Cyrillic, and others.
- Language Family: Including Germanic, Afro-Asiatic, and Indo-Aryan families.
Key Insights: The "Flip-Flop" Regime
One of the most surprising findings is the discovery of a "flip-flop" moment in model performance. Contrary to the assumption that pruning always degrades performance, the researchers found that for many low-resource languages, performance actually increased at moderate sparsity levels (typically between 50% and 70%) compared to the original dense model. In total, 26 out of the 40 languages studied showed improved F1 scores under these moderate pruning regimes.
This suggests that for multilingual models, the dense configuration may actually be suboptimal for specific tasks, and that removing redundant parameters can help the model specialize more effectively for underrepresented languages.
Compression as a Regularizer for Robustness
Beyond raw accuracy, the paper explores how compression affects robustness—the model's ability to handle unexpected changes in input. To test this, the authors used "entity mention replacement," swapping out names or locations in the test set with different entities of the same type.
This test revealed a stark contrast between dense and pruned models:
- Dense models often overfit to specific entities seen during training. When these entities were swapped, the F1 scores of dense models plummeted toward zero, indicating they were relying on memorized strings rather than linguistic context.
- Pruned models (specifically those using partial pruning) recovered significant performance. Pruning appears to act as a regularizer, forcing the model to ignore specific memorized weights and instead rely on the broader context to identify entities.
The Criticality of Embedding Layers
A vital nuance in these findings is the role of embedding layers. The robustness benefits and the performance gains in low-resource languages largely vanish if the embedding layers are pruned. This suggests that the "knowledge" required for cross-lingual transfer and robustness is stored differently than the weights required for standard task accuracy. Keeping the embeddings dense while sparsifying the transformer blocks allows the model to retain its foundational linguistic mappings while benefiting from the regularizing effects of pruning in the processing layers.
Conclusion and Future Work
This research reframes model compression not just as a tool for efficiency, but as a mechanism for improving model quality in multilingual settings. By carefully managing sparsity, developers can create models that are both smaller and more robust than their dense counterparts, particularly for low-resource languages.
However, the authors note several limitations. The study focused on mBERT and the NER task; it remains to be seen if these properties hold for larger models like XLM-R or for generative tasks such as translation. Additionally, the noise inherent in the WikiAnn dataset means that while the trends are clear, the absolute performance metrics should be interpreted with caution. Future work will involve deeper architectural analysis to understand exactly why the model becomes more robust and how these findings can be applied to the next generation of foundation models.
Summary updated