Subspace Bayesian Inference for Scalable Deep Learning

76 concepts52 papers1 proposal

Deep learning models are notoriously overconfident. When presented with out-of-distribution data, a standard neural network will often output incorrect predictions with high certainty. Bayesian Deep Learning (BDL) offers a mathematically elegant remedy: instead of learning a single fixed point estimate for the network's weights, we learn a posterior probability distribution over the weights, $p(w|\mathcal{D})$. By averaging predictions across this distribution, the model naturally expresses uncertainty when encountering unfamiliar inputs.

However, scaling Bayesian inference to modern neural networks faces a severe scalability crisis. If a model contains $D$ parameters, computing or storing a full posterior covariance matrix requires $\mathcal{O}(D^2)$ memory, and operations like matrix inversion scale at $\mathcal{O}(D^3)$ time complexity. For modern architectures with millions or billions of parameters, full-parameter Bayesian inference is computationally impossible. As Andrew Gordon Wilson argues in The Case for Bayesian Deep Learning, accounting for model uncertainty is crucial for robust generalization, yet full-parameter perturbations remain too expensive for practical deployment.

To resolve this bottleneck, a powerful paradigm has emerged: Subspace Bayesian Inference. Instead of performing inference over the massive, unconstrained parameter space $\mathbb{R}^D$, we restrict our probabilistic calculations to a highly compressed, low-dimensional subspace $\mathbb{R}^d$ (where $d \ll D$). This shift fundamentally changes the computational geometry of uncertainty quantification.


The Intuition: Intrinsic Low Dimensionality

Why can we get away with ignoring $99.9\%$ of the parameter space? The core intuition lies in the geometry of the loss landscapes of deep networks. The directions of parameter variation that actually matter for fitting data and capturing epistemic uncertainty are highly redundant and confined to a tiny fraction of the total dimensionality.

Mathematically, we can define the high-dimensional weight vector $w \in \mathbb{R}^D$ as a projection from a small latent coordinate vector $\theta \in \mathbb{R}^d$:

$$w = w_0 + P\theta$$

where $w_0$ is a central pivot weight vector (such as a pre-trained or maximum a posteriori solution) and $P \in \mathbb{R}^{D \times d}$ is an orthonormal projection matrix defining the subspace.

This formulation was systematically evaluated by Pavel Izmailov et al. in Subspace Inference for Bayesian Deep Learning. The authors demonstrated a surprising property of deep neural networks: the construction of the parameter subspace itself is far more critical for predictive performance and calibration than the specific choice of the Bayesian algorithm applied within it. A well-chosen subspace allows simple inference methods to match or exceed the performance of exhaustive sampling across the full unconstrained space. In fact, Daniel Dold et al. showed in Bayesian Semi-structured Subspace Inference that subspaces spanned by fewer than 16 dimensions can capture nearly the same posterior quality as exhaustive full-space sampling methods.


Mechanisms of Subspace Construction

To make this approach work, we must intelligently construct the projection matrix $P$. The literature converges on three primary strategies:

1. Data-Driven and Trajectory-Based Projections

One way to find the most informative directions is to observe the path the model takes during optimization. By saving snapshots of the weights during the late stages of standard training (e.g., during Stochastic Weight Averaging), we can perform Principal Component Analysis (PCA) or Singular Value Decomposition (SVD) on these weight trajectories to extract the principal directions of variation.

Patryk Marszałek et al. leveraged this in Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA, demonstrating that meaningful epistemic uncertainty can be captured within an extremely small subspace derived from the principal components of pre-trained weights. Similarly, A N M Nafiz Abeer et al. showed in Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design that compressing millions of parameters into a tiny "active subspace" allows for efficient variational inference in complex generative models without retraining the base architecture.

2. Structural and Parameter-Efficient Adapters

Instead of finding a subspace post-hoc, we can hardcode a low-rank subspace directly into the model's architecture using Low-Rank Adaptation (LoRA). By freezing the massive pre-trained weight matrices and injecting small, trainable low-rank matrix factors ($A$ and $B$), we can restrict our Bayesian treatment exclusively to these lightweight adapters.

Standard Weight Update:  [  W (Frozen)  ] + [      ΔW (Full Rank)      ]
                                                 ↓
Subspace Adapter (LoRA): [  W (Frozen)  ] + [ Matrix A ] × [ Matrix B ]
                                                 ↓
Stochastic Bottleneck:   [  W (Frozen)  ] + [ Matrix A ] × [ Stochastic Z ] × [ Matrix B ]

This strategy has democratized uncertainty quantification for Large Language Models. Adam X. Yang et al. introduced this concept via Laplace-LoRA in Bayesian Low-rank Adaptation for Large Language Models, proving that weight-space distributions can be estimated with minimal computational overhead on models with billions of parameters. Emre Onal et al. expanded on this in Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models, showing that applying stochastic weight averaging to these adapter spaces yields excellent calibration with negligible memory overhead.

To push the parameter tax even lower, Amir Hossein Rahmati et al. proposed a three-matrix factorization in C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models, which inserts a tiny, stochastic bottleneck matrix between standard low-rank update matrices, isolating the random parameters to an absolute minimum.

3. Learned Latent Representations

A third paradigm uses generative models to learn the underlying distribution of neural network weights. Rather than relying on linear projections like PCA, the authors of Encoding the Latent Posterior of Bayesian Neural Networks for Uncertainty Quantification (Emanuel Aldea et al.) trained a Variational Autoencoder (VAE) to encode the weight space. By performing Bayesian inference within the non-linear latent space of the VAE, they captured complex weight correlations that traditional diagonal approximations completely miss, matching the performance of expensive deep ensembles.


Why It Works: Computational and Mathematical Tractability

Restricting inference to a subspace alters the underlying mathematics of Bayesian calculations, converting intractable bottlenecks into efficient operations.

Linearizing Variational Complexity

In Variational Inference (VI), we approximate the true posterior using a tractable distribution (like a Gaussian) and optimize its parameters by maximizing the Evidence Lower Bound (ELBO). In a full parameter space, optimizing a Gaussian covariance matrix requires managing $\mathcal{O}(D^2)$ parameters.

By projecting the variational parameters onto a lower-dimensional orthonormal subspace, Edward Challis et al. proved in Concave Gaussian Variational Approximations for Inference in Large-Scale Bayesian Linear Models that the computational complexity drops from quadratic to linear relative to the original dimensionality. This allows Gaussian variational approximations to scale seamlessly to tens of thousands of dimensions.

Subspace Regularization

When optimizing a BNN via variational inference, we must continuously calculate the Kullback-Leibler (KL) divergence between our variational posterior and a prior distribution. If this calculation requires projecting back up to the full parameter space $\mathbb{R}^D$, the cost remains high.

Yibin Wang et al. solved this in BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models by demonstrating that the KL divergence can be calculated entirely within the low-rank subspace of the model adapters. This mathematical shortcut eliminates the need to map distributions back to the full weight space, drastically lowering the computational cost of regularizing learned distributions.

Bypassing Matrix Inversion via Geometry Approximation

For Laplace approximations, the core challenge is computing and inverting the curvature matrix (the Hessian or the Generalized Gauss-Newton matrix). In Accelerated Linearized Laplace Approximation for Bayesian Deep Learning, Zhijie Deng et al. proved that a very low-rank approximation of the posterior geometry—viewed through the lens of Neural Tangent Kernels (NTKs)—is sufficient to capture essential uncertainty information. By leveraging low-rank structures, they reduced high-dimensional matrix inversion problems into small, computationally efficient operations without sacrificing calibration quality.


Real-World Impact and Applications

By decoupling uncertainty quantification from model size, subspace inference makes Bayesian reliability practical across several demanding domains:

  • Active Learning for LLMs: In alignment tasks like Reinforcement Learning from Human Feedback (RLHF), selecting which data points require human labeling is a major bottleneck. Luckeciano Carvalho Melo et al. applied subspace inference to preference modeling in Deep Bayesian Active Learning for Preference Modeling in Large Language Models, enabling efficient data selection on models with hundreds of billions of parameters by placing lightweight Bayesian adapters on top of frozen weights.
  • Online Learning and Neural Bandits: Traditional neural network bandits suffer from catastrophic forgetting or require massive data replay buffers. Gerardo Duran-Martin et al. bypassed this in Efficient Online Bayesian Inference for Neural Bandits by recursively updating an Extended Kalman Filter within a low-dimensional parameter subspace. This allowed the network to maintain a running posterior distribution with constant memory and time complexity.
  • Inverse Problems and Scientific Computing: In high-dimensional nonlinear inverse problems, Tiangang Cui et al. showed in Likelihood-informed dimension reduction for nonlinear inverse problems that the posterior can be decomposed into a complex, non-Gaussian component within a "Likelihood-Informed Subspace" (LIS) and a simple, data-independent Gaussian component in the remaining directions. This allows researchers to focus expensive Markov chain Monte Carlo (MCMC) sampling exclusively on the few parameter directions actually constrained by the data.

Ultimately, subspace Bayesian inference works because it respects the intrinsic geometry of deep learning. By focusing probabilistic machinery only on the directions that drive model behavior, it delivers the safety and calibration of Bayesian inference at a fraction of the computational cost.

Key Foundational Work

Speeding up Magnetic Resonance Image Acquisition by Bayesian Multi-Slice Adaptive Compressed Sensing (2009) — contributed 1 concept elements to this direction.
Sofomore: Combined EEG Source and Forward Model Reconstruction (2009) — contributed 1 concept elements to this direction.
Last stand before WMAP: Cosmological parameters from lensing, CMB, and galaxy clustering (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_rank_posterior_approximation_e11be773,
  title = {low-rank posterior approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-rank-posterior-approximation/rdr_e11be7732f3747fd89ab6977199400b8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_rank_posterior_approximation_e11be773_v1,
  title = {low-rank posterior approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-rank-posterior-approximation/rdr_e11be7732f3747fd89ab6977199400b8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake