Deterministic Initialization for Bayesian Deep Learning

20 concepts15 papers1 proposal

Bayesian Neural Networks (BNNs) offer a mathematically elegant solution to one of deep learning's greatest vulnerabilities: knowing what they do not know. By treating network weights as probability distributions rather than fixed point estimates, BNNs naturally quantify predictive uncertainty.

Yet, in practice, training a BNN from scratch often feels like an exercise in frustration. While standard deterministic networks converge reliably using modern optimizers, their Bayesian counterparts frequently suffer from severe optimization failures, mode collapse, or poor predictive accuracy.

To bridge this gap, a powerful paradigm has emerged: deterministic-to-probabilistic initialization. Instead of training probabilistic models from an uninformed random starting point, researchers use a pre-trained deterministic model to "warm-start" the Bayesian optimization process. This simple shift fundamentally alters the geometry of the loss landscape, turning an otherwise intractable optimization problem into a highly stable refinement process.

The Optimization Crisis of Uninformed Priors

To understand why deterministic initialization works, we must first look at why traditional Bayesian training fails.

When training a BNN via Variational Inference, the goal is to optimize the Evidence Lower Bound (ELBO). The ELBO balances two competing forces: the data likelihood (how well the model fits the training data) and the KL divergence (a penalty that prevents the variational posterior distribution from straying too far from a chosen prior distribution).

Typically, researchers initialize BNNs with an uninformed prior, such as a standard normal distribution $\mathcal{N}(0, I)$. This choice introduces two severe pathologies in high-dimensional parameter spaces:

  1. Posterior Collapse: At the start of training, the model's weights are highly chaotic, meaning the data likelihood gradients are noisy and uninformative. However, the gradient of the KL term is perfectly clean, pulling the variational distribution directly toward the zero-centered prior. The model quickly collapses into the prior, losing its ability to learn meaningful representations.
  2. Gradient Divergence: If the prior is overly restrictive or highly confident, the optimization path becomes incredibly rugged. As noted by Martin Marek et al. in Can a Confident Prior Replace a Cold Posterior?, initializing a BNN from scratch under these conditions can cause the sampler to enter a divergence phase, where gradients point entirely away from the target class, causing training to fail completely.

The Intuition: Deterministic Networks as Geometric Scouts

Deterministic optimization is highly efficient at navigating high-dimensional loss landscapes to find deep, high-performance basins of attraction. The core insight of deterministic-to-probabilistic transfer is to use a standard neural network as a geometric scout.

Traditional BNN Training:
[Uninformed Prior: N(0, I)] ---> [Rugged Landscape] ---> Optimization Failure / Collapse

Deterministic-to-Probabilistic Training:
[Deterministic Pre-training] ---> [Locates High-Accuracy Basin] ---> [Initialize BNN Mean/Prior Here] ---> Stable Uncertainty Refinement

By first training a deterministic model via standard maximum a posteriori (MAP) or mean squared error optimization, we pinpoint a region in the weight space where the model already exhibits high predictive accuracy and well-developed feature representations.

As Ranganath Krishnan et al. demonstrate with the MOPED framework in Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes, setting the initial parameters of a Bayesian network based on these pre-trained weights bypasses the chaotic phase of early training. Because the model starts inside a high-performance basin, the data likelihood gradients are immediately stable and accurate, preventing the KL term from triggering posterior collapse.

This two-stage strategy separates the optimization workload into two distinct phases:

  1. Phase 1 (Deterministic): Learn what the features are (locate the center of the distribution).
  2. Phase 2 (Probabilistic): Learn how certain the model is about those features (spread the distribution out to capture variance).

Two Flavors of Transfer: Seeding Priors vs. Posteriors

Researchers implement this deterministic-to-probabilistic transfer primarily through two distinct mathematical mechanisms: empirical prior seeding and variational posterior initialization.

1. Empirical Prior Seeding

In this approach, the deterministic weights are used to define the prior distribution itself. Instead of a zero-centered Gaussian, the prior is centered around the converged deterministic weights.

The authors of Bayesian Deep Learning for Convective Initiation Nowcasting Uncertainty Estimation leverage this technique for high-dimensional meteorological forecasting, finding that empirical priors allow complex architectures to converge effectively where uninformed priors fail. By anchoring the prior to a known good solution, the model retains its baseline deterministic accuracy while safely exploring the surrounding parameter space to estimate variance.

2. Variational Posterior Initialization (Bayesian Fine-Tuning)

Alternatively, the deterministic weights can be used to initialize the mean of the variational posterior distribution, while keeping a standard prior. This is often referred to as Bayesian fine-tuning or refinement.

In BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning, Hao Zhang et al. show that initializing the posterior mean with converged deterministic weights places the model in a high-accuracy region from step one. This allows practitioners to turn a pre-trained model into a reliable BNN at a fraction of the computational cost of training from scratch.

A key benefit of this approach is that it can be applied selectively. For instance, in LiBRe: A Practical Bayesian Approach to Adversarial Detection, Zhijie Deng et al. initialize only the final layers of a pre-trained network as a Bayesian sub-module. This allows the system to inherit the rich feature representations of the deterministic backbone while gaining robust out-of-distribution and adversarial detection capabilities.

Universal Benefits Across Probabilistic Frameworks

The success of initializing probabilistic models from point estimates extends far beyond standard variational BNNs, showing up as a recurring theme across diverse machine learning domains:

Conclusion

Training a probabilistic model from scratch forces it to solve two incredibly difficult problems simultaneously: discovering meaningful features and mapping their underlying uncertainty. By breaking this process into a two-stage pipeline—using deterministic optimization to locate the optimal basin, and variational inference to map its width—deterministic-to-probabilistic initialization makes Bayesian deep learning practical, stable, and scalable.

Key Foundational Work

Improving the IBM Alignment Models Using Variational Bayes (2012) — contributed 1 concept elements to this direction.
Good Initializations of Variational Bayes for Deep Models (2019) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 26 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_deterministic_to_probabilistic_initializ_b08bc733,
  title = {deterministic-to-probabilistic initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deterministic-to-probabilistic-initialization/rdr_b08bc73384a1491aa899ec3f519ed942},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_deterministic_to_probabilistic_initializ_b08bc733_v2,
  title = {deterministic-to-probabilistic initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deterministic-to-probabilistic-initialization/rdr_b08bc73384a1491aa899ec3f519ed942/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake