Abstract
AdapterBias is a highly parameter-efficient adapter architecture designed to reduce the storage overhead of fine-tuning large Pre-trained Language Models (PLMs). By introducing a token-dependent representation shift—calculated as the outer product of a task-specific vector and a token-specific weight—AdapterBias achieves state-of-the-art performance on the GLUE benchmark with as few as 0.17 million trainable parameters per task. This approach represents a 40x reduction in parameters compared to standard adapters while maintaining competitive accuracy and demonstrating superior robustness in low-data regimes.
Introduction
Fine-tuning large-scale Pre-trained Language Models (PLMs) like BERT and RoBERTa has become the standard for achieving high performance across diverse Natural Language Processing (NLP) tasks. However, full fine-tuning requires storing a complete set of model parameters for every downstream task, leading to a significant "storage tax." While parameter-efficient methods like Adapters and BitFit have been proposed, they often either require millions of parameters or apply uniform shifts that do not account for the varying importance of individual tokens within a sentence.
Motivation: The Need for Token-Dependent Shifts
Existing efficient methods like BitFit only tune the bias terms of a model. While extremely lightweight, BitFit treats every token identically, applying the same shift regardless of a word's semantic or syntactic relevance to the task. The authors of AdapterBias argue that a model should prioritize task-critical words—such as "excellent" in sentiment analysis—over generic tokens like "the."
Methodology
AdapterBias is integrated into each transformer block, typically following the second feed-forward layer. Its architecture is defined by three primary components:
- Task-Specific Vector ($v$): A learnable vector that captures the general shift required for a specific downstream task.
- Linear Layer ($L_\alpha$): A lightweight layer that analyzes the input token representation to produce a scalar weight ($\alpha$).
- The Shift ($B$): The final bias added to the representation is computed as the outer product of the task vector and the token weight ($B = v \otimes \alpha^T$).
This mechanism allows the linear layer to act as a gate, modulating the task-specific shift based on the importance of the current token. To further enhance efficiency, the authors employ cross-layer parameter sharing and $L_0$ regularization to prune redundant weights.
Experimental Results
The researchers evaluated AdapterBias using the GLUE benchmark and SQuAD v1.0. The results demonstrate extreme parameter efficiency without sacrificing performance:
- Efficiency: Using a BERT-large backbone, AdapterBias achieved an average GLUE score of 81.2 with only 0.17M trainable parameters. In contrast, standard Houlsby Adapters require 7.14M parameters to achieve a nearly identical score (81.1).
- Performance: AdapterBias outperformed other efficient methods like LoRA (81.0) and maintained parity with BitFit (81.3) while offering a more nuanced, token-aware adaptation.
- Sparsity: Through $L_0$ regularization, the model successfully pruned up to 73.8% of the adapter's linear layer weights in tasks like CoLA with negligible impact on accuracy.
Discussion and Analysis
Analysis of the learned weights reveals that AdapterBias automatically "discovers" linguistic importance. In sentiment tasks, the model assigns significantly larger shifts to descriptive adjectives. In syntactic tasks (e.g., CoLA), the model focuses on reflexive pronouns, with peak activity observed in the 9th layer of BERT—a layer previously identified as a "sweet spot" for syntactic information.
Furthermore, AdapterBias shows remarkable robustness in low-data scenarios. Because it has fewer trainable parameters, it is less susceptible to overfitting than full fine-tuning, often yielding higher accuracy when training examples are scarce.
Conclusion
AdapterBias provides a minimalist yet powerful solution for parameter-efficient fine-tuning. By leveraging the outer product to generate high-dimensional, token-dependent shifts from a small set of parameters, it effectively bridges the gap between the extreme efficiency of bias-only tuning and the high performance of larger adapter modules. Future work may extend this token-dependent philosophy to other domains, such as computer vision.
Summary updated