Abstract
Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting large pre-trained models while only tuning a small number of parameters. They have been shown to be competitive with full model fine-tuning for many downstream tasks. However, prior work indicates that PEFTs may not work as well for machine translation (MT), and there is no comprehensive study showing when PEFTs work for MT. We conduct a comprehensive empirical study of PEFTs for MT, considering (1) various parameter budgets, (2) a diverse set of language-pairs, and (3) different pre-trained models. We find that ‘adapters’, in which small feed-forward networks are added after every layer, are indeed on par with full model fine-tuning when the parameter budget corresponds to 10% of total model parameters. Nevertheless, as the number of tuned parameters decreases, the performance of PEFTs decreases. The magnitude of this decrease depends on the language pair, with PEFTs particularly struggling for distantly related language-pairs. We find that using PEFTs with a larger pre-trained model outperforms full fine-tuning with a smaller model, and for smaller training data sizes, PEFTs outperform full fine-tuning for the same pre-trained model.
Introduction
In the era of Large Language Models, "full fine-tuning"—updating every single parameter of a model for a specific task—is becoming prohibitively expensive. For Neural Machine Translation (NMT), where supporting hundreds of language pairs is often required, storing a full copy of a billion-parameter model for every single pair is a storage and deployment nightmare.
Parameter-Efficient Fine-Tuning (PEFT) methods like Adapters or Prefix-tuning offer a solution: keep the massive "backbone" model frozen and only train a tiny fraction (often <1%) of new parameters. While these methods have been successful in classification tasks, their effectiveness in complex generation tasks like MT was poorly understood. This paper provides a comprehensive "stress test" of PEFTs for translation across different parameter budgets, language similarities, and model sizes.
Methods and Experimental Setup
The authors compare several prominent PEFT techniques:
- Adapters: Small bottleneck feed-forward networks inserted between existing transformer layers.
- Prefix-tuning: Adding learnable "prefix" vectors to the input of each layer.
- BitFit: Updating only the bias terms and LayerNorm weights of the model.
- X-attention Tuning: Updating only the cross-attention layers between the encoder and decoder.
The study evaluates these methods against two massive multilingual backbones: mBART (pre-trained on a denoising objective) and M2M-100 (pre-trained specifically for many-to-many translation). The researchers used a diverse set of 12 typologically different languages paired with English, sourced from TED Talks, IWSLT, and MultiParaCrawl.
Key Results and Discussion
The Impact of Parameter Budget and Pre-training
Across almost all budgets, Adapters emerged as the most robust PEFT method. At a 10% parameter budget, they were statistically on par with full fine-tuning. However, the effectiveness of these methods is heavily influenced by the nature of the pre-trained backbone.
As shown in Figure 1, there is a significant "Pre-training Gap." PEFTs work significantly better when the backbone model was pre-trained on the same task. M2M-100 (pre-trained for translation) reached 90% of full fine-tuning performance with far fewer parameters than mBART (pre-trained for denoising). If the model already "knows" how to translate, it requires fewer new parameters to specialize.
Language Distance and Scaling
Structural similarity between language pairs is a critical factor. When translating between very different languages (e.g., English to Korean), tiny PEFTs struggle as the model needs more capacity to bridge the linguistic gap.
Figure 5 illustrates that as language distance increases, the performance of low-budget PEFTs (like adapt-5 or bias) declines more sharply than higher-budget versions. Interestingly, the authors found that using a larger model with tiny adapters often outperforms a smaller model with full fine-tuning. This suggests that for deployment, it is more efficient to host one massive frozen backbone and swap out tiny adapter files for different languages.
PEFT as a Regularizer
In "low-resource" settings with very small datasets (fewer than 2,000 examples), full fine-tuning often overfits. In these cases, Adapters actually outperformed full fine-tuning because their constrained parameter count acted as a natural bottleneck, preventing the model from memorizing the small training set.
Conclusion
This study demonstrates that Adapters are a highly effective alternative to full fine-tuning for Machine Translation, particularly when paired with translation-specific pre-trained models. While performance decreases as the parameter budget shrinks—especially for distant language pairs—the efficiency and regularization benefits of PEFTs make them a compelling choice for large-scale NMT deployment. Future work remains to explore newer methods like LoRA and the performance of PEFTs on languages entirely absent from the pre-training data.
Summary updated