Abstract
Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pre-trained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.
Introduction
Large-scale Pre-trained Language Models (PLMs) such as BERT and RoBERTa have become the standard for solving complex Natural Language Processing (NLP) tasks. However, their massive size introduces a significant "storage tax." Fine-tuning a full model for every specific downstream task requires storing hundreds of millions of parameters per application, which is often impractical for large-scale deployments.
To mitigate this, researchers have turned to Adapters—small, trainable modules inserted into a frozen PLM. While standard adapters reduce the number of parameters compared to full fine-tuning, they can still be relatively heavy. A more recent alternative, BitFit, tunes only the bias terms of the model. While extremely efficient, BitFit applies the same static shift to every token in a sequence. This approach is fundamentally limited because it ignores the fact that different words carry different levels of importance for a specific task. For instance, in sentiment analysis, a word like "excellent" should arguably receive a more significant representation shift than a neutral word like "the."
The AdapterBias Architecture
This paper introduces AdapterBias, a lightweight adapter that implements a "token-dependent representation shift." Instead of a static bias, the shift is dynamically calculated based on the input token's hidden state.
The architecture relies on two primary components:
- A Task-Specific Shift Vector ($v$): A learned vector representing the general direction the task requires the representations to move.
- A Token-Dependent Weight ($\alpha$): A scalar produced by a small linear layer ($L_\alpha$) that analyzes the current token's hidden state.
The resulting bias $B$ for a sequence is calculated as the outer product: $B = v \otimes \alpha^T$. This allows the model to "turn up the volume" on specific tokens that are critical for the task at hand.
To further optimize efficiency, the authors explored two additional techniques:
- Cross-layer Parameter Sharing: Sharing the linear layer ($L_\alpha$) across all transformer blocks to minimize the parameter footprint.
- $L_0$ Regularization: A technique used during training to encourage sparsity in $L_\alpha$, effectively zeroing out unnecessary parameters without sacrificing performance.
Experimental Setup
The researchers evaluated AdapterBias across a wide range of standard NLP benchmarks to test its versatility and efficiency:
- GLUE Benchmark: Including tasks for sentiment (SST-2), linguistic acceptability (CoLA), similarity (STS-B, MRPC, QQP), and natural language inference (MNLI, QNLI, RTE).
- SQuAD v1.0: A reading comprehension dataset used to evaluate performance in data-intensive extractive QA scenarios.
- Base Models: The method was tested using BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large as the backbone architectures.
Key Results and Insights
Extreme Parameter Efficiency
AdapterBias achieves a remarkable reduction in trainable parameters. For BERT-large, AdapterBias requires only 0.17M trainable parameters. In comparison, standard Adapters require 7.14M, and LoRA (Low-Rank Adaptation) requires 0.39M. Despite tuning less than 0.05% of the total parameters, AdapterBias remains highly competitive.
Competitive Performance
On the GLUE benchmark, AdapterBias achieved an average score of 81.2, nearly matching the 82.2 achieved by full fine-tuning of BERT-large. This demonstrates that a simple linear transformation of the bias is sufficient to capture the nuances required for complex task adaptation.
Linguistic Sensitivity and the "Ninth Layer"
Qualitative analysis revealed that AdapterBias automatically learns linguistic relevance. In sentiment tasks, the model assigns higher weights to descriptive adjectives like "bad" or "wonderful." In syntactic tasks like CoLA, it focuses on reflexive pronouns (e.g., "himself").
Furthermore, the authors observed the "Ninth Layer" phenomenon in BERT-large: the 9th layer consistently received the most significant representation shifts for syntactic tasks. This aligns with existing literature suggesting that different layers of transformer models specialize in different types of linguistic information.
Sparsity and Robustness
By applying $L_0$ regularization, the authors reduced the parameters of the linear layer by an additional 17% on average with negligible performance loss. Additionally, AdapterBias proved robust in low-data regimes, often outperforming full fine-tuning when training examples were scarce.
Conclusion
AdapterBias demonstrates that complex bottleneck architectures are not always necessary for effective model adaptation. By introducing a token-dependent shift, the method provides a highly efficient alternative to standard adapters and BitFit. While a small performance gap remains compared to full fine-tuning, the massive reduction in storage requirements makes AdapterBias a compelling choice for multi-task deployments. Future work may explore the application of AdapterBias to generative tasks and the potential for automated layer-wise tuning to further optimize which parts of the model receive representation shifts.
Summary updated