Abstract
The current modus operandi in NLP involves downloading and fine-tuning pre-trained models consisting of millions or billions of parameters. Storing and sharing such large trained models is expensive, slow, and time-consuming, which impedes progress towards more general and versatile NLP methods that learn from and for many tasks. Adapters -- small learnt bottleneck layers inserted within each layer of a pre-trained model -- ameliorate this issue by avoiding full fine-tuning of the entire model. However, sharing and integrating adapter layers is not straightforward. We propose AdapterHub, a framework that allows dynamic "stitching-in" of pre-trained adapters for different tasks and languages. The framework, built on top of the popular HuggingFace Transformers library, enables extremely easy and quick adaptations of state-of-the-art pre-trained models (e.g., BERT, RoBERTa, XLM-R) across tasks and languages. Downloading, sharing, and training adapters is as seamless as possible using minimal changes to the training scripts and a specialized infrastructure. Our framework enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. AdapterHub includes all recent adapter architectures and can be found at https://AdapterHub.ml.
Introduction
The standard approach in Natural Language Processing (NLP) involves taking a massive pre-trained model, such as BERT or RoBERTa, and fine-tuning all its parameters for a specific task. While highly effective, this method has encountered a scaling wall. With models now reaching billions of parameters, every new task necessitates storing and sharing multi-gigabyte files. This "fine-tuning explosion" is computationally expensive, environmentally taxing, and creates a significant barrier to model sharing and multi-task composition.
The AdapterHub Framework
To address these challenges, we introduce AdapterHub, a framework built on top of the HuggingFace transformers library. AdapterHub centers on the use of Adapters: small bottleneck layers inserted between the existing layers of a pre-trained Transformer.
The framework provides a library to "stitch" these adapters into models with minimal code changes. During fine-tuning, the original weights of the large model remain frozen, and only the parameters of the tiny adapter modules are updated. Furthermore, AdapterHub.ml serves as a centralized repository where researchers can upload, search for, and download pre-trained adapters for specific tasks and languages.
Adapter Architectures
AdapterHub is designed to be architecture-agnostic, supporting various adapter implementations. The two primary architectures integrated are those proposed by Houlsby et al. (2019) and Pfeiffer et al. (2020a). These architectures differ primarily in their placement within the Transformer block—either after both the multi-head attention and feed-forward layers (Houlsby) or primarily after the feed-forward layer (Pfeiffer).
Evaluation and Key Results
The framework was validated using the GLUE (General Language Understanding Evaluation) benchmark across tasks such as NLI (RTE, QNLI, MNLI), similarity (MRPC, STS-B, QQP), and sentiment analysis (SST-2, CoLA), primarily using BERT-Base and RoBERTa-Base.
Performance and Efficiency
- On-Par Accuracy: Adapters consistently perform within 1% of full fine-tuning. Notably, on the RTE task, the Pfeiffer architecture outperformed full fine-tuning (70.8 vs 66.2).
- Extreme Storage Savings: Because over 99% of parameters are fixed, an adapter for BERT-Base can be as small as 0.9 MB, compared to 440 MB for a full model. Storing two full models takes the same space as approximately 125 different adapters.
- Modularity: Encapsulated adapters allow for "stacking." For instance, a language-specific adapter can be combined with a task-specific adapter to facilitate cross-lingual transfer without retraining the underlying system.
Limitations and Future Work
While highly efficient, adapters are currently tied to the specific pre-trained model they were trained on. Future work will investigate mapping adapters between different base models. Additionally, while storage is saved, the extra layers introduce a minor inference overhead (latency). The team aims to expand support for more diverse Transformer architectures and explore complex "fusion" methods for multi-task learning.
Summary updated