Abstract
Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this work, we first show that BNN+LV suffers from a serious form of non-identifiability: explanatory power can be transferred between the model parameters and latent variables while fitting the data equally well. We demonstrate that as a result, in the limit of infinite data, the posterior mode over the network weights and latent variables is asymptotically biased away from the ground-truth. Due to this asymptotic bias, traditional inference methods may in practice yield parameters that generalize poorly and misestimate uncertainty. Next, we develop a novel inference procedure that explicitly mitigates the effects of likelihood non-identifiability during training and yields high-quality predictions as well as uncertainty estimates. We demonstrate that our inference method improves upon benchmark methods across a range of synthetic and real data-sets.
Introduction
In high-stakes machine learning applications, such as medical diagnosis or autonomous driving, understanding the source of uncertainty is as critical as the prediction itself. Uncertainty is typically categorized into two types: epistemic uncertainty, which arises from a lack of data to identify the "true" underlying function, and aleatoric uncertainty, which stems from inherent stochasticity in the environment.
Bayesian Neural Networks with Latent Variables (BNN+LVs) are a powerful class of models designed to capture both. They utilize network weights ($W$) to represent epistemic uncertainty and auxiliary latent variables ($z$) to model complex, heteroscedastic (input-dependent) noise.
The Non-Identifiability Problem in BNN+LVs
The core challenge with BNN+LVs is non-identifiability. In these models, the latent noise variables $z_n$ are treated as additional inputs to the neural network alongside the observed data $x_n$. This structure creates a "signal-noise confusion": the model can fit the training data by either learning the true underlying function through the weights $W$ or by allowing the latent variables $z_n$ to "memorize" specific training points.
This pathology leads to an asymptotic bias in the joint posterior mode. Even with an infinite amount of data, standard inference methods may prefer "cheating" solutions where $z$ explains the signal, leading to poor generalization and misleading uncertainty estimates. Specifically, the model might over-estimate aleatoric uncertainty in regions with no data, rather than correctly identifying it as epistemic uncertainty.
Noise-Constrained Approximate Inference (NCAI)
To address this, the researchers propose Noise-Constrained Approximate Inference (NCAI), a two-step procedure designed to force the latent variables to behave like true noise rather than additional explanatory features.
1. Model-Satisfying Initialization
Standard random initialization often allows the latent variables to dominate early in training. NCAI begins by initializing the network weights using a deterministic neural network trained on the same dataset. This ensures the model captures the primary signal before the latent variables are introduced to explain the residuals.
2. Constrained Variational Inference
The standard Evidence Lower Bound (ELBO) objective is modified by introducing constraints that enforce the theoretical properties of noise. Using Lagrangian multipliers and differentiable proxies, the method requires the inferred latent variables to:
- Be independent of the inputs ($x$): Pearson correlation is used as a penalty to prevent $z$ from becoming a function of $x$.
- Follow the specified prior distribution: The Henze-Zirkler test is employed as a proxy to ensure the distribution of $z$ remains close to the Gaussian prior.
Experimental Evaluation
NCAI was evaluated against standard Mean-Field Variational Inference (MFVI) across a variety of benchmarks:
- Synthetic Datasets: Using standard heteroscedastic regression benchmarks (e.g., Goldberg, Williams, Heavy-Tail), the authors demonstrated that NCAI recovers ground-truth noise patterns more accurately than MFVI. For instance, on the "Heavy-Tail" dataset, NCAI achieved a test log-likelihood of -1.426, significantly outperforming MFVI's -1.867.
- Real-World UCI Datasets: On datasets like Lidar, Yacht Hydrodynamics, and Wine Quality, NCAI consistently provided better generalization and more reliable uncertainty intervals.
Qualitative analysis shows that while standard BNN+LVs often place high predictive mass in data-sparse regions (misidentifying epistemic uncertainty as aleatoric), NCAI maintains a tighter fit to the data distribution, correctly attributing uncertainty.
Conclusion and Limitations
NCAI effectively mitigates the non-identifiability of BNN+LVs by explicitly enforcing noise properties during inference. However, several challenges remain. The constrained objective is more difficult to optimize and requires careful tuning of the Lagrangian multipliers. Additionally, the use of linear correlation and the Henze-Zirkler test are heuristic proxies that may not fully capture independence in high-dimensional or highly non-linear settings. Future work will explore more robust independence measures and the theoretical consistency of the marginal posterior in these models.
Summary updated