Abstract
This paper investigates the "low-resource double bind"—the reality that languages with the least training data are often spoken in regions with the most severe computational and financial constraints. The authors discover that high levels of model pruning (up to 90% sparsity) not only maintain translation quality but can actually improve a model's ability to generalize to new domains by preventing it from "memorizing" noise in limited training sets.
The Problem: A Double Bind for NLP
In the current "bigger is better" era of AI, state-of-the-art Natural Language Processing (NLP) models are massive, making them expensive to train and deploy. While model compression (like pruning) is a standard solution, research has predominantly focused on high-resource languages like English or German.
The authors identify a low-resource double bind: low-resource languages (those with little digital data) are frequently spoken in areas where the cost of data and hardware is prohibitively high. This creates a gap where the populations who most need efficient models are the ones for whom compression is least understood.
Methodology
The researchers employed Magnitude Pruning, a technique where network weights with the smallest absolute values are removed. Their large-scale empirical study evaluated:
- Varying Sparsity: Models from 0% (dense) to 98% sparse.
- Data Regimes: A "Full" regime and a "Limited" regime (capped at 200k sentences) to simulate extreme scarcity.
- Granular Evaluation: Splitting tests into "Global" (frequent phrases) and "Random" (rare words/structures) to analyze what the model forgets.
- Human Evaluation: Verifying automated BLEU metrics with human raters.
The study focused on English translations into Yoruba, Hausa, Igbo (low-resource African languages) and German (high-resource baseline), primarily using the JW300 parallel corpus.
Key Insights: Sparsity as a Regularizer
A central finding is that in low-resource settings, pruning acts as a regularizer. While pruning is typically seen as a trade-off between size and quality, 90% sparse models often outperformed dense models on out-of-distribution (OOD) data (e.g., technical documentation or everyday phrases).
The authors argue that dense models have the capacity to "memorize" specific quirks and noise in small training sets (like religious jargon in JW300). Sparse models, lacking this excess capacity, are forced to learn more general linguistic patterns that generalize better across different domains.
Results and Performance
- The 90% Sweet Spot: Across all languages, models were remarkably tolerant of pruning up to 80-90%. Performance only drops significantly after 95% sparsity.
- Long-Tail Sensitivity: Pruning affects the "long tail" (infrequent or complex sentences) more severely than frequent, "prototypical" sentences at extreme sparsity levels.
- Human Verification: Human evaluation confirmed that for languages like Igbo and Hausa, 90% sparse models were often rated slightly higher than dense models for common phrases.
Conclusion and Future Work
The study demonstrates that high-level pruning is not only viable but potentially beneficial for low-resource machine translation. Future work includes investigating how pruning affects multilingual models and its potential to reduce "hallucinations" by preventing the memorization of training noise.
Summary updated