TL;DR
This paper tackles the computational cost of model pruning for Neural Machine Translation (NMT) by introducing a novel data pruning strategy. By identifying and retaining only the "hard-to-learn" training examples, they significantly reduce training time and memory requirements for movement pruning, achieving competitive translation quality.
What Problem Does This Paper Address?
Large pre-trained language models have revolutionized NLP, but their sheer size leads to high memory and computational demands, especially during fine-tuning and deployment. Model pruning techniques aim to reduce these costs by removing redundant parameters. However, the process of pruning, particularly methods like movement pruning, is itself computationally intensive, often requiring multiple cycles of training, pruning, and retraining. This makes efficient pruning challenging in resource-constrained environments. The authors aim to address this by reducing the amount of training data needed for effective pruning, specifically for NMT. They ask: how much data is truly necessary for efficient fine-pruning, and can we identify a metric to select the most informative training examples to prune the rest?
Methods
The core contribution is a data pruning algorithm designed to work in conjunction with movement pruning for NMT.
- Pruning Metric: They leverage training dynamics to identify informative examples. Specifically, they use the cross-entropy (CE) loss of individual training instances as an intrinsic metric and the BLEU score of individual examples as an extrinsic metric. The CE loss is calculated by fine-tuning a pre-trained model on the complete dataset once.
- Data Pruning Strategy: Based on the computed scores, they employ a Top-K strategy to select the "hard-to-learn" examples. This is compared against other strategies like Bottom-K (easy-to-learn), random selection, and stratified sampling.
- Movement Pruning: The selected subset of data is then used for movement pruning. Movement pruning is an unstructured pruning method that considers weight changes (movement) during fine-tuning, allowing weights to be pruned if they shrink, not just if their magnitude is small. This method is well-suited for transfer learning as it combines fine-tuning and compression.
The key novelty lies in using training dynamics (specifically CE loss) to prune the dataset before applying model pruning, thereby reducing the computational burden of the latter.
Datasets
The experiments were conducted on two standard NMT datasets:
- WMT16 En-Ro: Romanian-to-English translation.
- WMT16 En-Tr: Turkish-to-English translation.
These datasets are used for both training and evaluation. The paper notes that En-Tr is a more challenging dataset due to linguistic differences. No new benchmarks were introduced.
Key Insights and Ideas
- Hard-to-learn examples are crucial: Selecting hard-to-learn examples (Top-K strategy) consistently yields better BLEU scores compared to selecting easy-to-learn, random, or stratified examples for NMT fine-tuning.
- Data pruning significantly speeds up fine-tuning: By using pruned datasets, the number of convergence steps and thus the overall training time for movement pruning is substantially reduced. For instance, pruning 60% of the data for En-Tr resulted in a 48.9% faster fine-tuning process.
- Vocabulary reduction is minimal with Top-K: The Top-K strategy leads to the least reduction in vocabulary size among the pruning strategies, which might explain its superior performance.
- Pruning scores are transferable: Subsets pruned using one model (T5) can be effectively used for fine-tuning another model (BART), suggesting that the learned rankings are dataset-specific rather than model-specific.
- Unpruned model is better for ranking: Using an unpruned model to compute the initial cross-entropy scores for ranking is superior to using a pruned model, as the latter can lead to a less discriminative distribution of scores.
Key Results
- Performance: The Top-K data pruning strategy, when combined with movement pruning, achieved comparable or better BLEU scores than using the full dataset, even with significant data pruning (e.g., up to 60-80% reduction). For example, on En-Ro, Top-K with 60% data pruning achieved a BLEU score of 25.62, very close to the full dataset performance.
- Efficiency: Fine-tuning with 40% of the data for En-Tr was 48.9% faster than training with 80% of the data. For En-Ro, reducing data from 80% to 20% (Top-K) reduced convergence steps by 29.6% with only a 2.22% drop in BLEU.
- Vocabulary Impact: Top-K pruning resulted in the least decrease in vocabulary size, with a reduction of 83,911 unique tokens for En-Ro at 80% pruning, compared to much larger reductions for other strategies.
Limitations and Future Work
- Limited Datasets: The study was conducted on only two NMT datasets (En-Ro, En-Tr). Future work should evaluate the approach on a wider variety of NMT datasets, including those with different resource levels, noise, or domains.
- Cheaper Pruning Metrics: The current method requires an initial fine-tuning run to compute scores. Exploring cheaper, potentially self-supervised metrics that don't require a full training run could further reduce the initial cost.
- Ethical Impact: The data pruning strategies might lead to the under-representation of certain groups, potentially introducing bias. Future work should include comprehensive explainability and fairness evaluations of models trained on pruned data.
- Other Downstream Tasks: The authors suggest extending the pruning strategy to other downstream NLP tasks beyond NMT.
- In-depth Analysis: Further analysis of the pruned subsets and their impact on model behavior is recommended.
Summary updated