Abstract
Model compression is typically viewed as a necessary trade-off for deploying large-scale language models on resource-constrained devices. However, this study reveals that compression can act as a "secret weapon" for multilingual models. By evaluating mBERT on Named Entity Recognition (NER) across 40 languages, we find that moderate pruning (50-70% sparsity) can actually improve performance for low-resource languages and significantly boost model robustness against noisy or perturbed inputs. These results challenge the conventional "bigger is always better" narrative and suggest that sparsity can serve as an effective regularizer for cross-lingual generalization.
Introduction
As large language models (LLMs) continue to scale, they encounter the "curse of multilinguality": the challenge of representing an increasing number of languages within a fixed parameter budget. To facilitate deployment, researchers often employ compression techniques such as pruning or quantization. While aggregate performance metrics often remain stable, previous work suggests that compression can disproportionately affect "sub-populations," particularly low-resource languages. This paper maps the complex trade-offs between model scale, multilinguality, and compression to determine if high performance can be maintained across all languages in a compressed regime.
Methodology
The study focuses on Named Entity Recognition (NER) using mBERT as the base model. The authors apply Iterative Magnitude Pruning (IMP) during the fine-tuning stage, comparing two primary strategies:
- Partial Pruning: Pruning only the dense transformer layers while maintaining the full embedding table.
- Pruning with Embeddings: Pruning all model parameters, including the embedding layers.
Datasets and Robustness Evaluation
The researchers utilized WikiAnn, a multilingual NER dataset covering 40 languages across various resource levels, scripts, and language families. To evaluate robustness, they introduced a Perturbation Benchmark using entity mention replacement. By swapping entities with others from the same language, script, or family, they tested whether models relied on memorizing specific words or truly understood contextual cues.
Analysis of Compression Effects
The "Flip-Flop" Moment and Performance Gains
A key finding is that moderate sparsity often leads to performance improvements, particularly for low-resource languages. For 26 out of 40 languages, pruning to 50-70% sparsity resulted in higher F1 scores than the original dense baseline. This "Flip-Flop" suggests that the extra capacity in dense models may introduce noise for low-resource tasks, whereas pruning acts as a regularizer that improves generalization.
Compression as a Robustness Booster
Dense models often exhibit "lazy" behavior, relying on memorized entities rather than context. When these entities are perturbed, dense model performance frequently collapses. In contrast, sparse models—forced to operate with fewer parameters—appear to learn more robust contextual representations. Partial pruning at various levels was found to recover a significant portion of the performance lost to perturbations.
The Critical Role of Embeddings
While pruning dense layers enhances robustness, the authors found that pruning the embedding layers is detrimental. This indicates that a dense representation of the vocabulary is essential for handling out-of-distribution examples, even when the reasoning layers are sparse. Furthermore, training on multiple languages simultaneously (multilingualism) acts as a buffer, helping to offset the quality loss typically associated with high compression levels.
Conclusion and Future Work
This research demonstrates that model compression is not merely a deployment necessity but can actively improve model quality and robustness in multilingual settings. However, the benefits are bounded; at extreme sparsity (98%+), performance for low-resource languages degrades sharply, potentially widening the digital divide.
Future work should address the limitations of this study, including the reliance on the "silver standard" WikiAnn dataset and the focus on mBERT. Extending these findings to larger models like XLM-R and generative tasks such as translation will be crucial for a more comprehensive understanding of these "intriguing properties."
Summary updated