AdapterHub: A Framework for Adapting Transformers

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych

Abstract

AdapterHub is an open-source framework and repository designed to facilitate the sharing and utilization of adapters—modular neural network layers for large pre-trained models. By freezing the base model and training only these small bottleneck layers, AdapterHub achieves state-of-the-art performance on diverse tasks with significantly reduced storage and compute requirements. This modular approach effectively transforms massive language models into customizable platforms for efficient multi-task learning and research collaboration.

Introduction

The current standard in Natural Language Processing (NLP) involves taking a massive pre-trained model, such as BERT or RoBERTa, and fine-tuning all of its parameters for a specific task. While highly effective, this approach has several critical flaws. First, it is storage-inefficient; maintaining separate copies of a 500MB+ model for dozens of tasks is impractical. Second, it often leads to "catastrophic forgetting," where the model loses knowledge from previous tasks during new training. Finally, the lack of portability for these massive models hinders collaboration and research, particularly in low-resource settings.

The AdapterHub Framework

AdapterHub addresses these challenges by leveraging adapters: small bottleneck layers inserted between the existing layers of a pre-trained Transformer. During training, the original parameters of the large base model are kept frozen, and only the tiny adapter weights are updated.

Overview of the AdapterHub ecosystem and workflow.

Built on top of the HuggingFace transformers library, AdapterHub uses "MixIns" to allow users to add adapters with as little as two lines of code. A centralized website (AdapterHub.ml) acts as a repository where users can upload, search for, and download pre-trained adapters for specific tasks and languages, creating an "App Store" for Transformer models.

Technical Architecture

Adapters are typically implemented as bottleneck layers consisting of a down-projection, a non-linearity, and an up-projection. The framework supports various adapter architectures (such as those proposed by Houlsby et al. or Pfeiffer et al.) and allows users to define custom placements, residual connections, and activation functions via a configuration file.

Architecture of a Transformer layer with integrated adapters.

Because the base model remains frozen, adapters are encapsulated. This modularity allows researchers to stack adapters (e.g., a language adapter followed by a task adapter) or fuse them together without different tasks interfering with each other's weights.

Key Results and Efficiency

The framework was evaluated using the GLUE (General Language Understanding Evaluation) benchmark. Results demonstrate that adapters achieve performance on-par with full fine-tuning. For example, on the SST-2 sentiment task, BERT-Base with adapters scored 92.2-92.8, compared to 92.6 for full fine-tuning.

The primary advantage of this approach is extreme efficiency:

  • Parameter Reduction: Adapters typically require updating less than 1% of the total parameters while maintaining state-of-the-art accuracy.
  • Storage Savings: An adapter for BERT-Base can be as small as 0.9 MB, compared to the 440 MB required for a full model. For a "Large" transformer, a compressed adapter requires only ~3.2 MB, whereas the full model requires ~1.2 GB. This allows for storing over 100 task-specific adapters in the space usually occupied by just two fully fine-tuned models.

Conclusion

AdapterHub provides a scalable and computationally sustainable path for NLP research. By reducing the need to re-train and store massive models, it offers a more "green" approach to model adaptation. While current adapters are model-dependent (e.g., a BERT adapter cannot yet be used in RoBERTa), future work will focus on cross-model mapping and exploring complex "adapter fusion" techniques for advanced multi-task reasoning.

Summary updated

Abstract

The current modus operandi in NLP involves downloading and fine-tuning pre-trained models consisting of millions or billions of parameters. Storing and sharing such large trained models is expensive, slow, and time-consuming, which impedes progress towards more general and versatile NLP methods that learn from and for many tasks. Adapters -- small learnt bottleneck layers inserted within each layer of a pre-trained model -- ameliorate this issue by avoiding full fine-tuning of the entire model. However, sharing and integrating adapter layers is not straightforward. We propose AdapterHub, a framework that allows dynamic "stitching-in" of pre-trained adapters for different tasks and languages. The framework, built on top of the popular HuggingFace Transformers library, enables extremely easy and quick adaptations of state-of-the-art pre-trained models (e.g., BERT, RoBERTa, XLM-R) across tasks and languages. Downloading, sharing, and training adapters is as seamless as possible using minimal changes to the training scripts and a specialized infrastructure. Our framework enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. AdapterHub includes all recent adapter architectures and can be found at AdapterHub.ml
BibTeX
@inproceedings{pfeiffer2020adapterhub,
  author = {Jonas Pfeiffer and Andreas Rücklé and Clifton Poth and Aishwarya Kamath and Ivan Vulic and Sebastian Ruder and Kyunghyun Cho and Iryna Gurevych},
  title = {{AdapterHub: A Framework for Adapting Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-demos.7},
}
Figure