Bias-Only Fine-Tuning for Foundation Models

This direction is split into other directions: bias-only fine-tuning. Its last version stays readable and citable here.

Version 1 of 194 papers130 conceptsfrozen 2026-10-04

Fine-tuning a billion-parameter model usually requires massive computational resources and storage. However, researchers have discovered a surprising shortcut: you can achieve state-of-the-art performance by updating only the bias terms of a pre-trained network, leaving the massive weight matrices entirely frozen. This approach, often referred to as BitFit, modifies as little as 0.1% of the total parameters while matching or even exceeding the performance of full fine-tuning.

The Intuition: Steering Instead of Rebuilding

To understand why bias-only tuning works, we must distinguish between feature extraction and feature selection. The large weight matrices in a pre-trained Transformer act as a "skeleton" or a "Dominant Winning Ticket," as described by Zhuocheng Gong in Finding the Dominant Winning Ticket in Pre-Trained Language Models. These weights have already learned how to extract complex semantic and visual features from vast amounts of data.

Updating only the bias terms acts as a "steering" mechanism. As Shuailei Ma explains in Understanding the Multi-modal Prompts of the Pre-trained Vision-Language Model, these additive offsets function as a learnable dataset bias. They don't change the fundamental logic of the model; instead, they adjust the activation thresholds of the neurons. By shifting these thresholds, the model can prioritize specific pre-existing features (like foreground objects) that are relevant to the new task without "corrupting" the general-purpose knowledge stored in the weights.

Efficiency and the Privacy Advantage

The most immediate benefit of bias-only tuning is efficiency. In BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, Elad Ben Zaken demonstrated that updating only the biases in specific components, like the attention query vectors, captures the majority of performance gains while minimizing the trainable parameter count to roughly 0.04%.

This sparsity is particularly transformative for Differential Privacy (DP). Standard private fine-tuning is notoriously slow and memory-intensive because it requires storing large activation tensors to compute per-sample gradients. Zhiqi Bu and Yu-Xiang Wang introduced DP-BiTFiT in Differentially Private Bias-Term Fine-tuning of Foundation Models, showing that because bias gradients do not depend on the input activations of their own layer, the memory overhead is slashed by 8x and training speed increases by up to 30x.

Robustness and the Few-Shot Regime

In scenarios where data is scarce, full fine-tuning often leads to "catastrophic forgetting" or overfitting to spurious correlations. Bias-only adaptation serves as a powerful regularizer. Robert L. Logan IV found in Cutting Down on Prompts and Parameters that BitFit allows models to succeed with "null prompts"—simple concatenations of input—eliminating the need for brittle prompt engineering.

This robustness is critical for specialized domains:

  • Deepfake Detection: Yuxiang Xu uses bias-only learning in BPL to ensure models learn generalizable discrepancies between real and synthetic images rather than overfitting to the "fingerprints" of a specific generative model.
  • Medical Imaging: In Pixels Under Pressure, Zahra TehraniNasab notes that bias-only adaptation preserves the high-fidelity feature extractors of diffusion models while adjusting activation thresholds for diagnostic-grade synthetic images.
  • Safety Steering: Zonghan Yang demonstrated in UDDIA that optimizing only bias terms at inference time can effectively detoxify and debias language generation without damaging the model's underlying utility.

Beyond Transformers: Diffusion and State Space Models

The success of bias-only tuning has prompted its expansion into other architectures. Enze Xie proposed DiffFit in DiffFit: Unlocking Transferability of Large Diffusion Models, which adapts large-scale diffusion models to new resolutions and domains by updating just 0.12% of parameters. Similarly, for State Space Models (SSMs) like Mamba, Wonjun Kang developed State-offset Tuning in State-offset Tuning, which applies a trainable bias at every timestep to provide a uniform task-specific signal without adding computational latency.

Even for architectures that lack native bias parameters, the strategy remains viable. Researchers like Zhiqi Bu suggest injecting zero-initialized bias terms into existing layers. This simple modification allows any model to benefit from the efficiency and stability of bias-only adaptation, proving that the most effective way to teach an old model new tricks is often the smallest possible nudge.

Papers in this version

Showing 60 of 94 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bias_only_fine_tuning_for_foundation_mod_6716db02,
  title = {Bias-Only Fine-Tuning for Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bias-only-fine-tuning-for-foundation-models/rdr_6716db025a3b4fecb7e471a998dabfd4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bias_only_fine_tuning_for_foundation_mod_6716db02_v1,
  title = {Bias-Only Fine-Tuning for Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bias-only-fine-tuning-for-foundation-models/rdr_6716db025a3b4fecb7e471a998dabfd4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}