Towards a Unified View of Parameter-Efficient Transfer Learning

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig

Abstract

Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning (PETL) methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art PETL methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary. Through comprehensive empirical studies, we utilize the unified view to identify important design choices and instantiate new, more effective fine-tuning methods like the MAM Adapter.

Introduction

The standard paradigm in Natural Language Processing involves taking a massive pre-trained language model (PLM) and fine-tuning it for specific downstream tasks. While powerful, this "full fine-tuning" approach is increasingly unsustainable. Storing a full copy of a model with hundreds of billions of parameters for every single task is computationally and financially expensive.

To address this, researchers have developed Parameter-Efficient Transfer Learning (PETL). These techniques, such as Adapters, Prefix Tuning, and LoRA, freeze the majority of the pre-trained weights and only update a small fraction of parameters. However, because these methods were developed independently, the field lacked a clear understanding of why they work or how they relate. This paper provides that missing link, offering a unified framework to compare and improve PETL architectures.

A Unified Framework for PETL

The core contribution of this work is a mathematical deconstruction of popular PETL methods into a shared design space. The authors identify four primary dimensions that define how these methods modify a Transformer layer:

  1. Functional Form: This refers to the specific architecture used to compute the parameter update. For example, Adapters typically use a bottleneck neural network with a non-linearity, while LoRA uses a simple linear low-rank projection.
  2. Modified Representation: This dimension specifies which part of the Transformer is being altered—the Attention module (as in Prefix Tuning) or the Feed-Forward Network (FFN) (as in standard Adapters).
  3. Insertion Form: This describes how the new parameters are integrated into the existing architecture. They can be added sequentially (one after another) or in parallel to the original layers.
  4. Composition Function: This is the mathematical operation used to combine the new update with the original hidden state. Options include simple addition, gated addition, or scaled addition (a technique popularized by LoRA).
The unified framework illustrates how different PETL methods like Adapters, Prefix Tuning, and LoRA can be viewed as modular modifications to the standard Transformer architecture.

By analyzing methods through these dimensions, the authors made a surprising discovery: Prefix Tuning is essentially a Parallel Adapter. Mathematically, it functions as a bottleneck layer applied in parallel to the attention heads using a gated composition function. This insight allows researchers to "mix and match" the best components from different methods.

The MAM Adapter

Using the unified framework, the authors identified that parallel insertion and scaled composition are generally superior to sequential insertion and simple addition. They also found that while modifying Attention is efficient for tiny parameter budgets, modifying the FFN is more effective when more parameters are available.

Combining these insights, they proposed the MAM Adapter (Mix-And-Match Adapter). The MAM Adapter uses Prefix Tuning for the attention layers and Scaled Parallel Adapters for the FFN layers. This hybrid approach leverages the strengths of both methods to achieve maximum efficiency and performance.

Experimental Results

The authors evaluated the MAM Adapter and other PETL methods across four diverse benchmarks: XSum (summarization), WMT 2016 (translation), MNLI (inference), and SST2 (sentiment analysis). They specifically highlighted the importance of complex generation tasks like XSum, as simpler classification tasks often fail to reveal the performance gaps between efficient methods and full fine-tuning.

Key findings include:

  • Parity with Full Fine-Tuning: The MAM Adapter matched the performance of full fine-tuning across all tasks. On XSum, it achieved a 21.90 ROUGE-2 score (vs. 21.94 for full fine-tuning) while tuning only 6.7% of parameters.
  • Extreme Efficiency: On classification tasks like MNLI, the MAM Adapter reached full fine-tuning accuracy using only 0.5% of the pre-trained parameters.
  • Design Superiority: The study confirmed that parallel adapters consistently outperform sequential ones, and that scaling the adapter output (the LoRA composition style) significantly boosts performance.

Conclusion

This research demystifies the landscape of parameter-efficient transfer learning by providing a unified structural view. By breaking down methods into modular design dimensions, the authors were able to identify the most effective components and create the MAM Adapter—a state-of-the-art PETL method that offers the performance of full fine-tuning with a fraction of the storage and parameter costs. This work provides a clear roadmap for future research into making large-scale AI more accessible and efficient.

Summary updated

Abstract

Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.
BibTeX
@inproceedings{he2022towards,
  author = {Junxian He and Chunting Zhou and Xuezhe Ma and Taylor Berg-Kirkpatrick and Graham Neubig},
  title = {{Towards a Unified View of Parameter-Efficient Transfer Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=0RDcd5Axok},
}

Loading related content...

Figure