The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation

Orevaoghene Ahia, Julia Kreutzer, Sara Hooker

Abstract

A "bigger is better" explosion in the number of parameters in deep neural networks has made it increasingly challenging to make state-of-the-art networks accessible in compute-restricted environments. Compression techniques have taken on renewed importance as a way to bridge the gap. However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets. In this work, we instead consider the impact of compression in a data-limited regime. We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. This is a common setting for NLP for low-resource languages, yet the trade-offs in performance are poorly studied. Our work offers surprising insights into the relationship between capacity and generalization in data-limited regimes for the task of machine translation. Our experiments on magnitude pruning for translations from English into Yoruba, Hausa, Igbo and German show that in low-resource regimes, sparsity preserves performance on frequent sentences but has a disparate impact on infrequent ones. However, it improves robustness to out-of-distribution shifts, especially for datasets that are very distinct from the training distribution. Our findings suggest that sparsity can play a beneficial role at curbing memorization of low frequency attributes, and therefore offers a promising solution to the low-resource double bind.

Introduction

In the field of Natural Language Processing (NLP), the prevailing "bigger is better" paradigm has led to the development of massive neural networks. While these models achieve state-of-the-art performance, their computational requirements often make them inaccessible in resource-constrained environments. This creates what we call the low-resource double bind: a situation where communities speaking low-resource languages lack both the extensive training data needed for robust models and the affordable compute power required to deploy them.

The low-resource double bind: languages with lower data availability often face higher relative costs for internet access.

While model compression techniques like pruning are widely used to reduce model size, their impact has primarily been evaluated on high-resource languages. This study investigates the trade-offs of pruning in data-limited regimes, specifically for translation into African languages.

Methods

The study focuses on Magnitude Pruning, a technique that removes the weights with the smallest absolute values during training to create a sparse network.

  • Architecture: Standard Transformer-base models (~60M parameters).
  • Sparsity Levels: Tested from 0% (dense) to 98% sparse.
  • Languages: English to Yoruba (yo), Hausa (ha), Igbo (ig), and German (de) as a high-resource control.
  • Training Regimes: Comparison between "Full" data and "Limited" data (subsampled to 200k sentences) to isolate the effects of dataset size.
  • Evaluation: Performance was measured using BLEU scores, human evaluation, and analysis of "typical" (Global) versus "atypical" (Random) sentences.

Key Results

Resilience and the 90% Sparsity Sweet Spot

The experiments reveal that models are remarkably resilient to pruning. For all studied languages, BLEU scores remained stable up to 80-90% sparsity, with sharp declines only occurring at extreme levels (95-98%). Human evaluations corroborated this, often finding 90% sparse models indistinguishable from their dense counterparts.

Impact of magnitude pruning on BLEU scores across different languages and test sets.

Sparsity as a Regularizer

A surprising insight is that pruning acts as a regularizer in low-resource settings. Small datasets often contain noisy or idiosyncratic artifacts that models tend to overfit. By reducing model capacity, pruning forces the network to prioritize frequent, important patterns over the memorization of rare, low-frequency attributes.

The Long-Tail Sacrifice vs. OOD Robustness

While pruning preserves performance on frequent sentences, it disproportionately affects the "long-tail"—infrequent or atypical sentences. However, this reduction in memorization leads to improved Out-of-Distribution (OOD) robustness. Sparse models trained on limited data consistently outperformed dense models when tested on domains distinct from the training set (e.g., technical documentation or everyday speech), suggesting that sparsity helps the model generalize beyond its specific training domain.

Conclusion

The findings suggest that sparsity is a promising solution to the low-resource double bind. It not only reduces computational costs but also improves generalization by curbing the memorization of noise in small datasets. Future research should explore how these effects interact with other compression methods like quantization and the fine-tuning of large-scale pre-trained models.

Summary updated

Abstract

A "bigger is better" explosion in the number of parameters in deep neural networks has made it increasingly challenging to make state-of-the-art networks accessible in compute-restricted environments. Compression techniques have taken on renewed importance as a way to bridge the gap. However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets. In this work, we instead consider the impact of compression in a data-limited regime. We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. This is a common setting for NLP for low-resource languages, yet the trade-offs in performance are poorly studied. Our work offers surprising insights into the relationship between capacity and generalization in data-limited regimes for the task of machine translation. Our experiments on magnitude pruning for translations from English into Yoruba, Hausa, Igbo and German show that in low-resource regimes, sparsity preserves performance on frequent sentences but has a disparate impact on infrequent ones. However, it improves robustness to out-of-distribution shifts, especially for datasets that are very distinct from the training distribution. Our findings suggest that sparsity can play a beneficial role at curbing memorization of low frequency attributes, and therefore offers a promising solution to the low-resource double bind.
BibTeX
@inproceedings{ahia2021low,
  author = {Orevaoghene Ahia and Julia Kreutzer and Sara Hooker},
  title = {{The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=BLqZ0_L4Mmq},
}
Figure