Abstract
AdapterHub is an open-source framework and repository designed to facilitate the sharing and utilization of adapters—modular neural network layers for large pre-trained models. By freezing the base model and training only these small bottleneck layers, AdapterHub achieves state-of-the-art performance on diverse tasks with significantly reduced storage and compute requirements. This modular approach effectively transforms massive language models into customizable platforms for efficient multi-task learning and research collaboration.
Introduction
The current standard in Natural Language Processing (NLP) involves taking a massive pre-trained model, such as BERT or RoBERTa, and fine-tuning all of its parameters for a specific task. While highly effective, this approach has several critical flaws. First, it is storage-inefficient; maintaining separate copies of a 500MB+ model for dozens of tasks is impractical. Second, it often leads to "catastrophic forgetting," where the model loses knowledge from previous tasks during new training. Finally, the lack of portability for these massive models hinders collaboration and research, particularly in low-resource settings.
The AdapterHub Framework
AdapterHub addresses these challenges by leveraging adapters: small bottleneck layers inserted between the existing layers of a pre-trained Transformer. During training, the original parameters of the large base model are kept frozen, and only the tiny adapter weights are updated.
Built on top of the HuggingFace transformers library, AdapterHub uses "MixIns" to allow users to add adapters with as little as two lines of code. A centralized website (AdapterHub.ml) acts as a repository where users can upload, search for, and download pre-trained adapters for specific tasks and languages, creating an "App Store" for Transformer models.
Technical Architecture
Adapters are typically implemented as bottleneck layers consisting of a down-projection, a non-linearity, and an up-projection. The framework supports various adapter architectures (such as those proposed by Houlsby et al. or Pfeiffer et al.) and allows users to define custom placements, residual connections, and activation functions via a configuration file.
Because the base model remains frozen, adapters are encapsulated. This modularity allows researchers to stack adapters (e.g., a language adapter followed by a task adapter) or fuse them together without different tasks interfering with each other's weights.
Key Results and Efficiency
The framework was evaluated using the GLUE (General Language Understanding Evaluation) benchmark. Results demonstrate that adapters achieve performance on-par with full fine-tuning. For example, on the SST-2 sentiment task, BERT-Base with adapters scored 92.2-92.8, compared to 92.6 for full fine-tuning.
The primary advantage of this approach is extreme efficiency:
- Parameter Reduction: Adapters typically require updating less than 1% of the total parameters while maintaining state-of-the-art accuracy.
- Storage Savings: An adapter for BERT-Base can be as small as 0.9 MB, compared to the 440 MB required for a full model. For a "Large" transformer, a compressed adapter requires only ~3.2 MB, whereas the full model requires ~1.2 GB. This allows for storing over 100 task-specific adapters in the space usually occupied by just two fully fine-tuned models.
Conclusion
AdapterHub provides a scalable and computationally sustainable path for NLP research. By reducing the need to re-train and store massive models, it offers a more "green" approach to model adaptation. While current adapters are model-dependent (e.g., a BERT adapter cannot yet be used in RoBERTa), future work will focus on cross-model mapping and exploring complex "adapter fusion" techniques for advanced multi-task reasoning.
Summary updated