Abstract
Parameter-efficient fine-tuning (PEFT) methods adapt large pre-trained models by updating only a small fraction of parameters. While successful in many NLP tasks, their efficacy in machine translation (MT) has been less clear. This study provides a comprehensive empirical analysis of PEFTs for MT across varying parameter budgets, language pairs, and pre-trained architectures. We find that adapters can match full fine-tuning performance at a 10% parameter budget. However, performance declines as the budget shrinks, particularly for linguistically distant language pairs. Notably, applying PEFT to a larger model often outperforms full fine-tuning of a smaller model, and PEFT serves as an effective regularizer for small datasets.
Introduction
As Neural Machine Translation (NMT) models scale to billions of parameters, the standard practice of "full fine-tuning"—updating every parameter for every new language pair—becomes computationally and logistically prohibitive. Parameter-Efficient Transfer Learning (PEFT) offers a solution by freezing the bulk of the model and training only a small set of additional or selected parameters. While PEFT methods like adapters and prefix-tuning have shown parity with full fine-tuning in classification tasks, their performance in complex sequence-to-sequence tasks like translation has historically been inconsistent. This research investigates the specific conditions under which PEFT succeeds in MT.
Methodology and Experimental Setup
The study evaluates four primary PEFT techniques against a baseline of Full Fine-Tuning (FT):
- Adapters: Small bottleneck layers inserted within transformer blocks.
- Prefix-tuning: Learnable vectors prepended to the input of each layer.
- BitFit+: A modified version of BitFit that tunes LayerNorm weights in addition to biases.
- X-attention Tuning: Updating only the cross-attention layers and associated LayerNorms.
These methods were tested across two distinct pre-trained backbones: mBART (trained via denoising) and M2M-100 (trained via many-to-many translation). The evaluation spanned 12 languages using datasets from TED Talks, MultiParaCrawl, and IWSLT, covering a range of typological distances and data sizes.
Key Findings
1. The Importance of the Pre-training Objective
A critical discovery is the "Pre-training Gap." PEFT methods perform significantly better when the base model is already pre-trained for translation (e.g., M2M-100) rather than general denoising (e.g., mBART). Denoising models like mBART seem to require more extensive parameter updates to "re-learn" the translation task, making them less amenable to low-budget PEFT.
2. Linguistic Distance and Parameter Budgets
The effectiveness of PEFT is closely tied to the relationship between the source and target languages. For closely related pairs (e.g., English to Italian), PEFT matches full fine-tuning even at very low budgets. However, for distantly related pairs (e.g., English to Korean), the performance gap widens significantly. In these cases, a higher parameter budget (closer to 10%) is required to achieve parity with full fine-tuning.
3. The "Big Model" Advantage
The research highlights a practical strategy for resource-constrained environments: it is often more effective to apply a tiny PEFT budget to a massive model than to fully fine-tune a smaller model. For instance, an M2M-1.2B model with only 0.03% tuned parameters frequently outperforms a fully fine-tuned M2M-484M model. This suggests that the inherent knowledge in larger models compensates for the limited number of trainable parameters.
4. PEFT as a Regularizer
In scenarios with extremely limited training data (fewer than 2,000 examples), PEFT methods actually outperform full fine-tuning. Full fine-tuning is prone to overfitting on small datasets, whereas the constrained parameter space of PEFT acts as a natural regularizer, leading to better generalization.
Results and Comparisons
Among the tested methods, Adapters emerged as the most robust, consistently outperforming prefix-tuning and BitFit across various budgets. At a 10% parameter budget (bottleneck dimension $b=1024$), adapters matched or exceeded full fine-tuning performance across all tested scenarios.
Conversely, prefix-tuning was found to be less efficient in practice than its parameter count suggests. While it uses few parameters, it increases the effective sequence length, leading to higher memory consumption and slower training speeds compared to adapters.
Conclusion and Future Work
This study demonstrates that PEFT is a viable alternative to full fine-tuning for MT, provided the parameter budget and base model are chosen carefully. While adapters at a 10% budget are a safe default, even smaller budgets are effective when using translation-specific base models or when dealing with low-resource, high-regularization scenarios.
Future research should extend these findings to modern Large Language Models (LLMs) at the 70B+ parameter scale and incorporate newer PEFT techniques like LoRA. Additionally, exploring the performance of PEFT on languages not seen during the initial pre-training phase remains a vital area for the development of truly global translation systems.
Summary updated