Bayesian Neural Networks (BNNs) have long been the "holy grail" of deep learning because they treat weights as probability distributions rather than fixed points. This allows models to quantify uncertainty—essentially knowing when they don't know an answer. However, training BNNs from scratch is notoriously difficult. They often suffer from "cold start" problems, where uninformative priors (like a standard Gaussian centered at zero) lead to optimization failures or significantly lower accuracy compared to standard models.
A growing body of research suggests a more practical path: instead of training a BNN from scratch, we should "warm-start" it using a pre-trained deterministic model. By using the weights of a successful deterministic network to initialize the mean of a Bayesian prior, researchers can bridge the gap between high-performance point-estimate models and robust probabilistic frameworks.
The Problem of Uninformative Priors
In standard Bayesian inference, we start with a prior—our belief about the weights before seeing any data. Traditionally, researchers use "vague" or uninformative priors, such as $w \sim \mathcal{N}(0, I)$. While mathematically convenient, Minyoung Kim and Timothy Hospedales note in BayesDLL that these zero-mean priors fail to leverage the rich representational knowledge already present in modern foundation models.
When a BNN starts with zero-centered weights, it must rediscover complex features (like edges in images or syntax in language) while simultaneously trying to learn the variance (uncertainty) of those features. This often leads to the "Bayesian penalty," where the stochastic nature of the weights acts as a regularizer that actually degrades the accuracy of the model. Álvaro Cartea and colleagues observe in GAPA that this penalty can be avoided by initializing the prior mean to the values of a pre-trained deterministic model, thereby maintaining the original maximum a posteriori (MAP) performance.
Empirical Bayes as a Bridge
The core mechanism for this transition is Empirical Bayes. In this setup, the parameters of the prior are not chosen arbitrarily but are estimated from the data—or in this case, from a pre-trained model.
Ranganath Krishnan and his team at Intel introduced MOPED (Model Priors with Empirical Bayes using DNN), which formalizes this by setting the prior mean to the weights of a pre-trained deterministic network. This provides an "informed" starting point in the high-dimensional loss landscape. Instead of wandering through a sea of possible weight configurations, the Bayesian optimization process starts at a high-performing mode and simply explores the "width" of that mode to determine uncertainty.
This strategy has proven essential for scaling BNNs to massive architectures. For example, Zhijie Deng and Xiao Yang developed BayesAdapter (arxiv 2020), which converts pre-trained ResNets into variational BNNs through a few rounds of inexpensive fine-tuning. This approach bypasses the complex local optima that typically plague variational inference when starting from scratch.
Stability and Convergence
Beyond accuracy, pre-training provides critical stability. In complex tasks like Convective Initiation Nowcasting, Da Fan and John S. Schreck found that using weights from a deterministic model as a prior significantly improved convergence in high-dimensional parameter spaces.
Similarly, in the realm of 3D vision, Luca Savant Aira and Enrico Magli use a deterministic pre-training phase for Stochastic Gaussian Splatting. This initialization establishes a reliable baseline for scene parameters, preventing the instabilities and "mode collapse" that often occur when training stochastic models from a random state.
Applications Across Domains
The "deterministic-to-Bayesian" pipeline is becoming a standard template for adding reliability to specialized AI tasks:
- Dialogue Generation: Jing Yang Lee uses the BODEB framework (arxiv 2023) to transform pre-trained language models into BNNs. By inheriting the linguistic competence of the pre-trained model, the Bayesian version can generate diverse responses without losing coherence.
- Adversarial Detection: LiBRe, developed by Zhijie Deng and Hang Su, refines only the final layers of a pre-trained network into a Bayesian ensemble. This allows the model to detect adversarial attacks while retaining the high standard accuracy of the original deterministic model.
- Medical Imaging: In Sparse Bayesian Networks, Herve Lombaert and Samuel Kadoury use deterministic pre-training to identify which weights are most "sensitive," converting only those into Bayesian distributions to save on computational costs.
Summary of Intuition
The success of this research direction stems from a simple realization: it is easier to add uncertainty to a model that already works than it is to learn both function and uncertainty simultaneously. By treating a pre-trained model as a "prior belief," researchers can leverage the massive computational investment already made in deterministic deep learning to create safer, more self-aware probabilistic systems.