Bottleneck Adapters for Parameter Efficient Fine Tuning

862 concepts676 papers

When adapting large pre-trained models to new downstream tasks, full-parameter fine-tuning presents significant challenges. Updating millions or billions of parameters is computationally expensive, requires massive storage for each individual task, and risks distorting the highly optimized, general-purpose feature representations learned during pre-training—a phenomenon known as catastrophic forgetting.

Bottleneck adapters solve this problem by keeping the entire pre-trained model backbone frozen and injecting a tiny number of trainable parameters into strategic locations throughout the network. By forcing task-specific adjustments through a low-dimensional "bottleneck," these modules efficiently steer the model's behavior without altering its foundational knowledge.

The Core Mechanism: Low-Rank Residual Shifting

The fundamental intuition behind a bottleneck adapter is dimensionality reduction acting as a regularizer. Instead of allowing downstream training to alter the high-dimensional representation space of a pre-trained model, the adapter forces the intermediate features to pass through a highly compressed subspace.

Mathematically, let $h \in \mathbb{R}^d$ represent the hidden feature vector output by an internal layer of a transformer (where $d$ is typically large, such as 1024 or 4096). A standard bottleneck adapter processes this vector through three sequential steps:

  1. Down-Projection: A trainable linear layer $W_{\text{down}} \in \mathbb{R}^{d \times r}$ compresses the hidden vector into a low-dimensional bottleneck space $r$, where $r \ll d$ (e.g., $r = 64$). This compression forces the adapter to isolate and capture only the most critical task-specific variations.
  2. Non-Linear Activation: A non-linear activation function $\sigma(\cdot)$, such as ReLU or GeLU, introduces the expressiveness required to model complex task-specific mappings.
  3. Up-Projection: A second trainable linear layer $W_{\text{up}} \in \mathbb{R}^{r \times d}$ maps the compressed representation back up to the original dimension $d$.

$$\Delta h = \sigma(h W_{\text{down}}) W_{\text{up}}$$

Crucially, this entire operation is bypassed by a residual (skip) connection. The final output of the adapter layer is added directly back to the original hidden state:

$$h_{\text{new}} = h + \Delta h$$

This residual connection is the secret sauce of the architecture. As pioneered by Ankur Bapna et al. in Simple, Scalable Adaptation for Neural Machine Translation, initializing the weights such that the adapter initially outputs zero ensures that the module functions as a near-identity mapping at the start of training. Consequently, the model begins training with its pre-trained capabilities perfectly intact, stably learning only the residual adjustments ($\Delta h$) needed for the downstream task.

Structural Placement: Sequential vs. Parallel

Where should these bottleneck modules live within the frozen architecture? The field has converged on two primary paradigms: sequential and parallel placement.

Sequential Placement

In a sequential configuration, adapters are inserted directly into the main computation path of each transformer block. The authors of Adapters Strike Back demonstrate that placing bottleneck modules specifically at the end of a transformer block—after the feed-forward network (FFN) and its corresponding residual connection—optimizes performance. In this position, the adapter acts as a post-hoc refinement step, operating on fully processed features to tune the representation space for the downstream task.

Parallel Placement

Alternatively, adapters can be placed in parallel with existing frozen layers. Yunlong Zhao et al. in Parameter-Efficient Transfer Learning for End-to-end Speech Translation explore parallel bottleneck adapters that sit alongside the frozen FFN layers. Instead of modifying the features sequentially, parallel adapters ingest the same input as the frozen layer and add their low-rank output to the frozen layer's output. This approach modulates internal representations without disrupting or delaying the sequential flow of the core architecture.

Controlled Feature Shifting

A natural question is how bottleneck adapters compare to other parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA). While LoRA applies low-rank updates directly to the internal weight matrices (such as attention projections), bottleneck adapters perform feature-space transformations.

This distinction matters when preserving broad semantic capabilities is critical. Alexey Kravets et al. in CLIP Adaptation by Intra-modal Overlap Reduction highlight that weight-bound low-rank methods can sometimes be too aggressive, potentially overwriting or degrading the model's underlying knowledge. In contrast, bottleneck modules provide a more controlled mechanism for feature shifting. Because they act as distinct, isolated layers, they shift feature distributions without altering the internal weights of the backbone, making them highly effective at refining representations while mitigating catastrophic forgetting.

Architectural Variations Across Modalities

The simplicity of the bottleneck adapter has allowed it to be adapted across diverse architectures, tasks, and modalities:

By isolating task-specific learning to a low-rank, residual bottleneck, adapters preserve the integrity of large foundation models while offering a flexible, scalable, and parameter-efficient pathway for downstream customization.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bottleneck_adapter_modules_330effb4,
  title = {bottleneck adapter modules},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bottleneck-adapter-modules/rdr_330effb445304c05a2c4c913e75feee3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_bottleneck_adapter_modules_330effb4_v1,
  title = {bottleneck adapter modules},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bottleneck-adapter-modules/rdr_330effb445304c05a2c4c913e75feee3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake