Last Layer Bayesian Inference

69 concepts51 papers1 proposal

Deep neural networks are remarkably capable, but they suffer from a dangerous flaw: they do not know what they do not know. Standard deterministic networks output a single prediction, often with extreme overconfidence, even when presented with completely unfamiliar or out-of-distribution (OOD) data.

Bayesian Neural Networks (BNNs) offer a mathematically rigorous solution. By treating network weights not as fixed points, but as probability distributions, BNNs capture epistemic uncertainty—the model's doubt about its own parameters. When a BNN is uncertain, its weight distributions yield highly variable outputs, signaling that the model is guessing.

However, full BNNs are notoriously expensive. Placing a distribution over millions or billions of parameters requires integrating over a massive, high-dimensional weight space. This makes training slow, memory consumption prohibitive, and real-time inference nearly impossible.

Last-layer Bayesian inference solves this dilemma through a simple, powerful compromise: keep the entire feature extraction backbone completely deterministic, and apply Bayesian inference exclusively to the final decision layers.


The Core Intuition: Decoupling Representation from Decision

To understand why last-layer Bayesian inference works, it helps to view a deep neural network as a two-part pipeline:

  1. The Feature Extractor: The early and intermediate layers compress high-dimensional inputs (like images or text) into a low-dimensional latent space.
  2. The Classifier/Regressor: The final linear layer draws decision boundaries or fits a surface across these latent features.

As Laurent Valentin Jospin et al. outline, we can treat the initial layers as a static, deterministic engine that builds representations, while localizing all probabilistic behavior at the output stage.

Input ──> [ Deterministic Layers ] ──> Latent Features ──> [ Bayesian Layer ] ──> Uncertain Output
               (Feature Extractor)                             (The Final Judge)

But does ignoring the uncertainty in the early layers ruin our confidence estimates? Crucially, no. As Zhengyuan Liu et al. demonstrate, restricting Bayesian sampling to the final decision layers provides nearly identical uncertainty estimation quality as fully Bayesian models.

When an OOD sample enters the network, the deterministic backbone still projects it into the latent space. Because the model has never seen data from this distribution, the resulting latent feature vector lands in an unmapped, sparse region of the feature space. A probabilistic final layer will have high variance in these unmapped regions, naturally producing a wide distribution of outputs. As Kajetan Schweighofer et al. show, the vast majority of predictive variance required to capture epistemic uncertainty can be successfully recovered through this late-stage feature recombination.


Shrinking the Parameter Space

The primary benefit of this approach is computational tractability. By shifting variational inference from a high-dimensional parameter space to a low-dimensional space, the mathematical bottlenecks of BNNs evaporate.

For instance, Yadi Wei et al. introduce a method that models the final output directly as a Gaussian distribution, bypassing weight-space sampling altogether. When weight-space modeling is used, the drastically smaller parameter footprint allows researchers to apply highly sophisticated posterior refinement techniques that would fail on full networks. Agustinus Kristiadi et al. leverage this exact property, demonstrating that a highly refined posterior on just the final layer parameters yields OOD detection that outperforms many full-network Bayesian approaches.

This efficiency makes it possible to scale uncertainty quantification to complex architectures and real-time applications where traditional BNNs are non-viable, such as deep retrieval systems (Daniel Cohen et al.) and semi-supervised learning frameworks (Charlotte Loh et al.).


Common Implementation Strategies

Several distinct methodologies have emerged to parameterize and train these probabilistic final layers:

1. Last-Layer Laplace Approximation

Instead of training a BNN from scratch, you can take a standard, pre-trained deterministic network and apply the Laplace approximation only to the weights of the final layer. This involves computing the Hessian of the loss function at the maximum a posteriori (MAP) estimate to fit a Gaussian distribution over the weights.

As Erik A. Daxberger et al. show in their work on the Laplace Redux, this turns a standard model into a Bayesian one post-hoc with almost zero additional training cost. To speed up inference even further, Agustinus Kristiadi et al. utilize a "Laplace Bridge" to analytically map these Gaussian logit distributions directly to Dirichlet output distributions, eliminating the need for expensive Monte Carlo sampling during the forward pass.

2. Variational Inference and Stochastic Heads

Another approach replaces the final linear classification head with a layer that learns a distribution over weights via Stochastic Variational Inference (SVI). During training, the network optimizes the Evidence Lower Bound (ELBO) solely for this head. Ling Shao et al. utilize this partial parameterization to capture weight uncertainty in domain generalization, forcing the model to remain cautious when encountering shifted domains.

3. Data-Dependent Priors

Rather than assuming a simple Gaussian prior for the final layer, recent work by Valentin Villecroze et al. introduces Last Layer Empirical Bayes (LLEB). This method uses Normalizing Flows to learn a data-dependent prior specifically for the network's last layer, matching the robust performance of deep ensembles without the massive computational burden of training multiple independent models.


Generalizing the Principle: Partial Stochasticity

While isolating uncertainty quantification to the final layer is the most common paradigm, the underlying principle—partial Bayesian parameterization—extends to other targeted subsets of a network. The core idea remains identical: identify the small fraction of parameters that contribute most to uncertainty or regularization, and leave the rest deterministic.

  • Sensitivity-Driven Selection: Rather than picking a specific layer, Zeinab Abboud proposes converting only the most "sensitive" weights across a network into Bayesian distributions, matching ensemble performance while making 95% of the parameters deterministic.
  • Variance-Driven Selection: In federated learning environments, Hui Chen et al. apply Laplace approximations exclusively to a minority subset of high-variance weights to achieve well-calibrated client models without overloading communication bandwidth.
  • Input-Layer Stochasticity: In domains with extreme input noise, such as speech recognition, researchers sometimes flip the script. Jiajun Deng et al. selectively apply Bayesian inference to the initial layers of a network to absorb acoustic variability right at the source.

Ultimately, last-layer and partial Bayesian methods demonstrate that uncertainty quantification is not an all-or-nothing proposition. By recognizing that deep networks separate representation learning from decision-making, we can place our probabilistic budgets exactly where they matter most.

Key Foundational Work

Hands-on Bayesian neural networks—A tutorial for deep learning users (2020) — contributed 3 concept elements to this direction.
Uncertainty Modeling for Machine Comprehension Systems using Efficient Bayesian Neural Networks (2020) — contributed 2 concept elements to this direction.
Know Where To Drop Your Weights: Towards Faster Uncertainty Estimation (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_last_layer_bayesian_inference_5b6da708,
  title = {last-layer Bayesian inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/last-layer-bayesian-inference/rdr_5b6da7085e46431385bf6f4f135efa72},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_last_layer_bayesian_inference_5b6da708_v1,
  title = {last-layer Bayesian inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/last-layer-bayesian-inference/rdr_5b6da7085e46431385bf6f4f135efa72/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake