AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks

Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, Hung-yi Lee

Abstract

AdapterBias is a highly parameter-efficient adapter architecture designed to reduce the storage overhead of fine-tuning large Pre-trained Language Models (PLMs). By introducing a token-dependent representation shift—calculated as the outer product of a task-specific vector and a token-specific weight—AdapterBias achieves state-of-the-art performance on the GLUE benchmark with as few as 0.17 million trainable parameters per task. This approach represents a 40x reduction in parameters compared to standard adapters while maintaining competitive accuracy and demonstrating superior robustness in low-data regimes.

Introduction

Fine-tuning large-scale Pre-trained Language Models (PLMs) like BERT and RoBERTa has become the standard for achieving high performance across diverse Natural Language Processing (NLP) tasks. However, full fine-tuning requires storing a complete set of model parameters for every downstream task, leading to a significant "storage tax." While parameter-efficient methods like Adapters and BitFit have been proposed, they often either require millions of parameters or apply uniform shifts that do not account for the varying importance of individual tokens within a sentence.

Motivation: The Need for Token-Dependent Shifts

Existing efficient methods like BitFit only tune the bias terms of a model. While extremely lightweight, BitFit treats every token identically, applying the same shift regardless of a word's semantic or syntactic relevance to the task. The authors of AdapterBias argue that a model should prioritize task-critical words—such as "excellent" in sentiment analysis—over generic tokens like "the."

Conceptual comparison of representation shifts: BitFit (left) applies a uniform shift to all tokens, whereas AdapterBias (right) applies larger, task-specific shifts to critical tokens like "bad" or "good."

Methodology

AdapterBias is integrated into each transformer block, typically following the second feed-forward layer. Its architecture is defined by three primary components:

  1. Task-Specific Vector ($v$): A learnable vector that captures the general shift required for a specific downstream task.
  2. Linear Layer ($L_\alpha$): A lightweight layer that analyzes the input token representation to produce a scalar weight ($\alpha$).
  3. The Shift ($B$): The final bias added to the representation is computed as the outer product of the task vector and the token weight ($B = v \otimes \alpha^T$).

This mechanism allows the linear layer to act as a gate, modulating the task-specific shift based on the importance of the current token. To further enhance efficiency, the authors employ cross-layer parameter sharing and $L_0$ regularization to prune redundant weights.

Architectural comparison between Houlsby adapters, BitFit, and the proposed AdapterBias module within a standard Transformer block.

Experimental Results

The researchers evaluated AdapterBias using the GLUE benchmark and SQuAD v1.0. The results demonstrate extreme parameter efficiency without sacrificing performance:

  • Efficiency: Using a BERT-large backbone, AdapterBias achieved an average GLUE score of 81.2 with only 0.17M trainable parameters. In contrast, standard Houlsby Adapters require 7.14M parameters to achieve a nearly identical score (81.1).
  • Performance: AdapterBias outperformed other efficient methods like LoRA (81.0) and maintained parity with BitFit (81.3) while offering a more nuanced, token-aware adaptation.
  • Sparsity: Through $L_0$ regularization, the model successfully pruned up to 73.8% of the adapter's linear layer weights in tasks like CoLA with negligible impact on accuracy.

Discussion and Analysis

Analysis of the learned weights reveals that AdapterBias automatically "discovers" linguistic importance. In sentiment tasks, the model assigns significantly larger shifts to descriptive adjectives. In syntactic tasks (e.g., CoLA), the model focuses on reflexive pronouns, with peak activity observed in the 9th layer of BERT—a layer previously identified as a "sweet spot" for syntactic information.

Furthermore, AdapterBias shows remarkable robustness in low-data scenarios. Because it has fewer trainable parameters, it is less susceptible to overfitting than full fine-tuning, often yielding higher accuracy when training examples are scarce.

Conclusion

AdapterBias provides a minimalist yet powerful solution for parameter-efficient fine-tuning. By leveraging the outer product to generate high-dimensional, token-dependent shifts from a small set of parameters, it effectively bridges the gap between the extreme efficiency of bias-only tuning and the high performance of larger adapter modules. Future work may extend this token-dependent philosophy to other domains, such as computer vision.

Summary updated

Abstract

Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pre-trained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.
BibTeX
@inproceedings{fu2022adapterbias,
  author = {Chin-Lun Fu and Zih-Ching Chen and Yun-Ru Lee and Hung-yi Lee},
  title = {{AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=3K-nQOPw24d},
}
Figure