Bayesian Inference in Low-Rank Parameter Subspaces

This direction is split into other directions: low-rank posterior approximation. Its last version stays readable and citable here.

Version 1 of 173 papers116 conceptsfrozen 2026-10-04

Bayesian Neural Networks (BNNs) are the gold standard for uncertainty quantification, but they have long suffered from a "scalability tax." To represent a distribution over weights rather than a single point estimate, traditional BNNs often double or triple memory requirements and computational costs. This overhead is prohibitive for modern architectures like Llama 3 or DINOv2, which contain billions of parameters.

A growing body of research suggests a way out: we do not need to be Bayesian about every weight. Instead, we can perform inference within a highly compressed, low-rank subspace of the model's parameters. This approach captures the essential "epistemic" uncertainty—the model's awareness of what it doesn't know—at a fraction of the cost.

The Low-Dimensional Reality of High-Dimensional Models

The core intuition behind this direction is that while neural networks live in a high-dimensional parameter space, the "interesting" parts of the posterior distribution—the directions where the data actually changes our prior beliefs—are remarkably low-dimensional.

As Olivier Zahm and colleagues argue in Certified dimension reduction in nonlinear Bayesian inverse problems, the transition from a prior to a posterior in complex problems is often a low-rank update. In other words, the data only provides significant information along a few specific directions. Wesley J. Maddox and Pavel Izmailov demonstrated this empirically with SWAG, showing that the uncertainty of deep networks concentrates within a low-dimensional subspace spanned by the SGD optimization trajectory A Simple Baseline for Bayesian Uncertainty in Deep Learning..

By restricting our probabilistic modeling to these critical directions, we can use sophisticated tools like the Laplace approximation or Variational Inference without the cubic scaling costs that usually kill BNN performance.

Bayesian Low-Rank Adaptation

The most prominent application of this idea is the marriage of Bayesian methods with Low-Rank Adaptation (LoRA). LoRA is already the standard for parameter-efficient fine-tuning, where only a tiny pair of low-rank matrices ($A$ and $B$) are trained while the rest of the model remains frozen.

Researchers like Adam X. Yang and Maxime Robeyns have shown that applying a Laplace approximation specifically to these LoRA parameters—a method called Laplace-LoRA—provides reliable calibration for LLMs with almost no extra compute Bayesian Low-rank Adaptation for Large Language Models.

Other frameworks take this further:

Why Subspace Inference Works

The effectiveness of these methods stems from a "bottleneck" effect. In C-LoRA, Amir Hossein Rahmati and co-authors insert a small, stochastic bottleneck matrix between standard low-rank update matrices. This design isolates the random parameters to a minimal subspace, ensuring that the "noise" required for Bayesian sampling doesn't destabilize the entire network.

This principle is pushed to the extreme in B-LoRA-XS, where Patryk Marszałek demonstrates that meaningful epistemic uncertainty can be captured in a subspace derived from the principal components of pre-trained weights. Even when the rank of the covariance matrix is severely restricted, the model remains well-calibrated Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA.

Beyond Large Language Models

While LLMs are a primary target, the benefits of low-rank Bayesian inference extend to any high-dimensional domain:

Summary of the Intuition

The shift toward low-rank Bayesian inference represents a move from "modeling the weights" to "modeling the variation." By acknowledging that most parameters in a deep network are redundant for the purposes of uncertainty, researchers have found that we can achieve the robustness of a full Bayesian ensemble by simply being probabilistic about a tiny, well-chosen fraction of the model. As Andrew Gordon Wilson argues, this isn't just a computational trick—it is a more natural way to handle the over-parameterization of modern AI The Case for Bayesian Deep Learning.

Papers in this version

Showing 60 of 73 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bayesian_inference_in_low_rank_parameter_94533d7a,
  title = {Bayesian Inference in Low-Rank Parameter Subspaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-inference-in-low-rank-parameter-subspaces/rdr_94533d7a238b4ad18a403812c074d4d1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bayesian_inference_in_low_rank_parameter_94533d7a_v1,
  title = {Bayesian Inference in Low-Rank Parameter Subspaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-inference-in-low-rank-parameter-subspaces/rdr_94533d7a238b4ad18a403812c074d4d1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}