When does Parameter-Efficient Transfer Learning Work for Machine Translation?

Ahmet Üstün, Asa Cooper Stickland

Abstract

Parameter-efficient fine-tuning (PEFT) methods adapt large pre-trained models by updating only a small fraction of parameters. While successful in many NLP tasks, their efficacy in machine translation (MT) has been less clear. This study provides a comprehensive empirical analysis of PEFTs for MT across varying parameter budgets, language pairs, and pre-trained architectures. We find that adapters can match full fine-tuning performance at a 10% parameter budget. However, performance declines as the budget shrinks, particularly for linguistically distant language pairs. Notably, applying PEFT to a larger model often outperforms full fine-tuning of a smaller model, and PEFT serves as an effective regularizer for small datasets.

Introduction

As Neural Machine Translation (NMT) models scale to billions of parameters, the standard practice of "full fine-tuning"—updating every parameter for every new language pair—becomes computationally and logistically prohibitive. Parameter-Efficient Transfer Learning (PEFT) offers a solution by freezing the bulk of the model and training only a small set of additional or selected parameters. While PEFT methods like adapters and prefix-tuning have shown parity with full fine-tuning in classification tasks, their performance in complex sequence-to-sequence tasks like translation has historically been inconsistent. This research investigates the specific conditions under which PEFT succeeds in MT.

Methodology and Experimental Setup

The study evaluates four primary PEFT techniques against a baseline of Full Fine-Tuning (FT):

  1. Adapters: Small bottleneck layers inserted within transformer blocks.
  2. Prefix-tuning: Learnable vectors prepended to the input of each layer.
  3. BitFit+: A modified version of BitFit that tunes LayerNorm weights in addition to biases.
  4. X-attention Tuning: Updating only the cross-attention layers and associated LayerNorms.

These methods were tested across two distinct pre-trained backbones: mBART (trained via denoising) and M2M-100 (trained via many-to-many translation). The evaluation spanned 12 languages using datasets from TED Talks, MultiParaCrawl, and IWSLT, covering a range of typological distances and data sizes.

Key Findings

1. The Importance of the Pre-training Objective

A critical discovery is the "Pre-training Gap." PEFT methods perform significantly better when the base model is already pre-trained for translation (e.g., M2M-100) rather than general denoising (e.g., mBART). Denoising models like mBART seem to require more extensive parameter updates to "re-learn" the translation task, making them less amenable to low-budget PEFT.

2. Linguistic Distance and Parameter Budgets

The effectiveness of PEFT is closely tied to the relationship between the source and target languages. For closely related pairs (e.g., English to Italian), PEFT matches full fine-tuning even at very low budgets. However, for distantly related pairs (e.g., English to Korean), the performance gap widens significantly. In these cases, a higher parameter budget (closer to 10%) is required to achieve parity with full fine-tuning.

3. The "Big Model" Advantage

The research highlights a practical strategy for resource-constrained environments: it is often more effective to apply a tiny PEFT budget to a massive model than to fully fine-tune a smaller model. For instance, an M2M-1.2B model with only 0.03% tuned parameters frequently outperforms a fully fine-tuned M2M-484M model. This suggests that the inherent knowledge in larger models compensates for the limited number of trainable parameters.

4. PEFT as a Regularizer

In scenarios with extremely limited training data (fewer than 2,000 examples), PEFT methods actually outperform full fine-tuning. Full fine-tuning is prone to overfitting on small datasets, whereas the constrained parameter space of PEFT acts as a natural regularizer, leading to better generalization.

Results and Comparisons

Among the tested methods, Adapters emerged as the most robust, consistently outperforming prefix-tuning and BitFit across various budgets. At a 10% parameter budget (bottleneck dimension $b=1024$), adapters matched or exceeded full fine-tuning performance across all tested scenarios.

Conversely, prefix-tuning was found to be less efficient in practice than its parameter count suggests. While it uses few parameters, it increases the effective sequence length, leading to higher memory consumption and slower training speeds compared to adapters.

Conclusion and Future Work

This study demonstrates that PEFT is a viable alternative to full fine-tuning for MT, provided the parameter budget and base model are chosen carefully. While adapters at a 10% budget are a safe default, even smaller budgets are effective when using translation-specific base models or when dealing with low-resource, high-regularization scenarios.

Future research should extend these findings to modern Large Language Models (LLMs) at the 70B+ parameter scale and incorporate newer PEFT techniques like LoRA. Additionally, exploring the performance of PEFT on languages not seen during the initial pre-training phase remains a vital area for the development of truly global translation systems.

Summary updated

Abstract

Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting large pre-trained models while only tuning a small number of parameters. They have been shown to be competitive with full model fine-tuning for many downstream tasks. However, prior work indicates that PEFTs may not work as well for machine translation (MT), and there is no comprehensive study showing when PEFTs work for MT. We conduct a comprehensive empirical study of PEFTs for MT, considering (1) various parameter budgets, (2) a diverse set of language-pairs, and (3) different pre-trained models. We find that 'adapters', in which small feed-forward networks are added after every layer, are indeed on par with full model fine-tuning when the parameter budget corresponds to 10% of total model parameters. Nevertheless, as the number of tuned parameters decreases, the performance of PEFTs decreases. The magnitude of this decrease depends on the language pair, with PEFTs particularly struggling for distantly related language-pairs. We find that using PEFTs with a larger pre-trained model outperforms full fine-tuning with a smaller model, and for smaller training data sizes, PEFTs outperform full fine-tuning for the same pre-trained model.
BibTeX
@inproceedings{ustun2022when,
  author = {Ahmet Üstün and Asa Cooper Stickland},
  title = {{When does Parameter-Efficient Transfer Learning Work for Machine Translation?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=sVuWRFrPn4},
}
Figure