BayesTune: Bayesian Sparse Deep Model Fine-tuning

Minyoung Kim, Timothy Hospedales

Fine-tuning massive foundation models (FMs) has become the standard workflow in AI. However, updating all billions of parameters for every new task is computationally expensive and prone to overfitting. While "sparse fine-tuning"—updating only a tiny fraction of parameters—often yields better results, practitioners have mostly relied on "gut feelings" or simple heuristics (like only updating biases) to decide which parameters to change.

In their NeurIPS 2023 paper, BayesTune: Bayesian Sparse Deep Model Fine-tuning, Minyoung Kim and Timothy Hospedales introduce a principled way to let the data tell us exactly which parameters matter.

The Parameter Selection Problem

The core challenge in Parameter-Efficient Fine-Tuning (PEFT) is the selection problem: out of millions of parameters, which ones should we "unfreeze"?

Existing methods usually fall into two camps:

  1. Heuristics: Manually choosing specific modules (like Adapters) or parameter types (like BitFit, which only updates biases).
  2. Approximations: Using second-order Taylor expansions or heavy approximations to guess parameter importance.

The authors argue these methods lack a solid theoretical foundation. There is no guarantee that a hand-picked module or a rough gradient approximation captures the true "importance" of a parameter for a specific downstream task.

The BayesTune Method

BayesTune treats fine-tuning as a Bayesian inference problem. Here is the core logic behind their approach:

  1. The Sparse Prior: They place a Laplace prior on every parameter $\theta_i$ of the foundation model. The mean of this prior is set to the pre-trained value. Crucially, the "scale" (width) of this distribution, $\lambda_i$, is also treated as a random variable.
  2. The Hyper-prior: They place a Gamma distribution on $\lambda_i$ that strongly encourages it to be small.
  3. The Logic: If a parameter $\theta_i$ is unimportant for the new task, the posterior $\lambda_i$ will stay very small (the prior dominates). If the parameter must change to solve the task, the data will force the posterior $\lambda_i$ to become large.
  4. Efficient Sampling: Instead of slow traditional Bayesian methods, they use Stochastic Gradient Langevin Dynamics (SGLD). This allows them to sample the posterior with a computational cost only about twice that of standard SGD.
  5. Two-Stage Execution:
    • Stage 1: Run SGLD to estimate the posterior means of the scales ($\hat{\lambda}_i$).
  6. Stage 2: Sort the parameters by $\hat{\lambda}_i$, keep the top $p\%$ as "updatable," freeze the rest, and perform standard fine-tuning.

Key Insights: The $\hat{\lambda}$-Plot

Unlike many PEFT methods that freeze entire layers, BayesTune operates at the individual parameter level. It discovered that for many tasks, the "important" parameters are actually distributed somewhat uniformly across layers, rather than being concentrated at the top or bottom.

By plotting the sorted posterior scales, the authors provide a visual "elbow" method to choose the optimal sparsity level. This removes the guesswork from deciding whether to update 0.5% or 10% of the model.

Candidate cut-off points based on sorted posterior scales

A clever implementation detail is the "noise discount factor" ($\gamma$) in the SGLD sampler. This helps the algorithm balance between finding the most likely parameters (the mode) and exploring the uncertainty (the mean).

Key Results

The authors validated BayesTune on two major fronts: NLP (fine-tuning RoBERTa-base on GLUE/SuperGLUE) and Vision (fine-tuning ViT-B/16 on VTAB-1K). BayesTune consistently outperformed both random selection and sophisticated heuristics.

Performance comparison on GLUE and SuperGLUE benchmarks
  • NLP Superiority: On GLUE/SuperGLUE, BayesTune achieved an average accuracy of 78.12%, outperforming the previous state-of-the-art (SAM) by a full percentage point. It took first place in 5 out of 7 tasks.
  • Vision Versatility: On the VTAB-1K benchmark, it achieved the #1 rank on 7 out of 19 diverse imaging datasets.
  • Efficiency: Compared to Variational Inference (another Bayesian approach), BayesTune was 1.7x faster and used 2.1x less GPU memory, making it practical for large-scale foundation models.

Limitations and Future Work

While BayesTune is a significant step forward, the authors note a few areas for growth:

  • Two-Stage Overhead: The requirement of a "Stage 1" sampling phase means it takes longer than a simple one-pass fine-tuning.
  • Hyperparameter Sensitivity: The "effective data size" ($\hat{N}$) and noise discount factor ($\gamma$) still require some grid-searching on a validation set.
  • Theoretical Gap: While the empirical results are strong, a rigorous theoretical link between the scale threshold and final generalization performance remains an open research question.

In summary, BayesTune moves sparse fine-tuning from the realm of "engineering hacks" into a principled statistical framework, proving that sometimes, the best way to tune a model is to let the data tell you exactly what it needs.

Summary updated

Abstract

Deep learning practice is increasingly driven by powerful foundation models (FM), pre-trained at scale and then fine-tuned for specific tasks of interest. A key property of this workflow is the efficacy of performing sparse or parameter-efficient fine-tuning, meaning that by updating only a tiny fraction of the whole FM parameters on a downstream task can lead to surprisingly good performance, often even superior to a full model update. However, it is not clear what is the optimal and principled way to select which parameters to update. Although a growing number of sparse fine-tuning ideas have been proposed, they are mostly not satisfactory, relying on hand-crafted heuristics or heavy approximation. In this paper we propose a novel Bayesian sparse fine-tuning algorithm: we place a (sparse) Laplace prior for each parameter of the FM, with the mean equal to the initial value and the scale parameter having a hyper-prior that encourages small scale. Roughly speaking, the posterior means of the scale parameters indicate how important it is to update the corresponding parameter away from its initial value when solving the downstream task. Given the sparse prior, most scale parameters are small a posteriori, and the few large-valued scale parameters identify those FM parameters that crucially need to be updated away from their initial values. Based on this, we can threshold the scale parameters to decide which parameters to update or freeze, leading to a principled sparse fine-tuning strategy. To efficiently infer the posterior distribution of the scale parameters, we adopt the Langevin MCMC sampler, requiring only two times the complexity of the vanilla SGD. Tested on popular NLP benchmarks as well as the VTAB vision tasks, our approach shows significant improvement over the state-of-the-arts (e.g., 1% point higher than the best SOTA when fine-tuning RoBERTa for GLUE and SuperGLUE benchmarks).
BibTeX
@inproceedings{kim2023bayestune,
  author = {Minyoung Kim and Timothy Hospedales},
  title = {{BayesTune: Bayesian Sparse Deep Model Fine-tuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=TRuqrVsmZK},
}
Figure