When fine-tuning a massive pre-trained model, the standard approach is to update all of its weight matrices. However, this full fine-tuning process is computationally expensive, requires massive storage for task-specific model copies, and risks corrupting the rich, general-purpose representations the model learned during pre-training.
An elegant alternative is to freeze every single weight matrix in the network and update only the additive bias terms. This approach, popularized as BitFit by Ben Zaken et al., modifies less than 0.1% of the total parameters. Yet, it consistently matches the performance of full fine-tuning across a wide array of tasks. To understand why such a minimalist modification works, we have to look at the geometry of neural network activations and the mechanics of backpropagation.
Shifting Distributions Without Distorting Features
To build intuition, consider a single linear layer within a neural network, which computes the transformation:
$$y = Wx + b$$
The weight matrix $W$ determines how input features interact, rotate, and scale to form a new representation space. When we fine-tune $W$, we fundamentally alter the coordinate system of this space. If the fine-tuning dataset is small, modifying $W$ easily warps the underlying semantic manifold, leading to overfitting or the loss of general knowledge.
In contrast, the bias vector $b$ acts as a pure additive offset. Modifying $b$ does not change how features interact or rotate; instead, it shifts the entire activation distribution along specific axes. In networks that use non-linearities like ReLU or GeLU, this shift changes which features are activated or suppressed. As noted by Ding et al. in their exploration of delta-tuning, bias-only adaptation acts as a highly efficient steering mechanism, leveraging the model's existing representational capacity rather than trying to construct new features from scratch.
The Geometry of Bias Adaptation
How can a simple vector addition match the expressive power of full weight updates? The answer lies in how pre-trained models organize information.
Gong et al. investigated this phenomenon by looking for "winning tickets" within pre-trained language models. They discovered a striking spatial alignment: the spatial orientation of the network's bias vectors naturally overlaps with the most influential, dominant dimensions of the weight matrices. Because the biases are already aligned with the axes that govern model adaptation, tuning the biases targets the exact same intrinsic dimensions that full fine-tuning would alter.
Furthermore, not all biases are created equal. Ben Zaken et al. demonstrated that performance gains can be highly localized, such as targeting the biases within the attention query vectors and the intermediate layers of multi-layer perceptrons (MLPs). Conversely, Hazarika et al. found that certain bias terms, such as the bias in the "Key" transformation of dot-product attention, are mathematically redundant. By selectively freezing these non-functional parameters, researchers can shrink the optimization space even further without sacrificing downstream accuracy.
Bypassing the Activation Memory Bottleneck
Beyond parameter efficiency, bias-only tuning unlocks a massive, often overlooked computational advantage during training: it eliminates the need to store intermediate activations.
In standard backpropagation, calculating the gradient of a weight matrix requires both the incoming error gradient ($\delta$) and the layer's input activations ($x$):
$$\nabla_W L = \delta x^T$$
Because $x$ depends on the forward pass, the training hardware must cache the activations of every single layer in memory until the backward pass reaches them. This activation memory dominates the storage overhead of training large models.
For a bias term, however, the gradient calculation is independent of the input activations:
$$\nabla_b L = \delta$$
As Zhiqi Bu demonstrated, restricting updates exclusively to bias terms means the training framework can immediately discard input activations during the forward pass. This mathematical property slashes memory consumption by up to 8x and accelerates training up to 30x. This makes bias-only tuning uniquely suited for strict training environments, such as training under Differential Privacy (DP) constraints or reducing communication payloads in federated learning systems, as shown by Zhang et al..
Preserving the Pre-Trained Manifold
Because bias-only tuning leaves the weight matrices untouched, it acts as an intrinsic regularizer. It prevents the model from overfitting to dataset-specific artifacts, making it highly effective for few-shot learning and out-of-distribution generalization.
For instance, in deepfake detection, models often overfit to the specific generator fingerprints present in the training data. Xu et al. showed that freezing the weights and optimizing only the biases preserves the robust semantic manifold of the underlying foundation model, forcing the network to learn generalizable discrepancies rather than spurious training artifacts.
Similarly, when adapting models to small datasets or novel domains, full fine-tuning can cause catastrophic forgetting. By restricting updates to encoder biases, Kim et al. prevented few-shot dense prediction models from memorizing limited support examples. In zero-shot and few-shot settings where models exhibit intrinsic class biases, Kang He demonstrated that updating only internal bias parameters successfully calibrates the model's predictions without degrading its core linguistic capabilities.
Generalizing Across Modalities and Tasks
While early work focused heavily on masked language models, bias-only adaptation has proven to be a universal mechanism across diverse architectures and modalities:
- Translation and Personalization: Early exploration by Simianer et al. and Michel et al. proved that updating only the final output layer's bias terms allows neural machine translation systems to rapidly adapt to user corrections and individual speaker styles without altering underlying syntactic representations.
- Vision-Language and Diffusion Models: The authors of CLIPFit achieved state-of-the-art vision-language adaptation by surgically updating biases in the text encoder's projection layers alongside normalization layers. For generative vision, Xie et al. (DiffFit) extended this paradigm to large diffusion models, enabling rapid domain and resolution adaptation.
- Image Restoration and Compressive Sampling: In unrolled architectures used for image reconstruction, Qin et al. showed that test-time bias updates function similarly to soft-thresholding operators. This allows models to quickly pivot to new degradation types (like unseen weather conditions) while maintaining structural stability, a property also leveraged by Kim et al..
Handling Bias-Free Architectures
A common modern challenge is that several state-of-the-art architectures omit bias terms entirely from their linear layers to optimize raw hardware performance. To apply bias-only tuning to these models, Zhiqi Bu proposed injecting zero-initialized bias parameters into the existing layers. Because these injected parameters start at zero, they do not alter the model's initial pre-trained behavior, yet they immediately open the door to the parameter and memory efficiencies of bias-only adaptation.
Key Foundational Work
Measuring Immediate Adaptation Performance for Neural Machine Translation (2019) — contributed 1 concept elements to this direction.
Extreme Adaptation for Personalized Neural Machine Translation (2018) — contributed 1 concept elements to this direction.