Learning Deep Bayesian Latent Variable Regression Models that Generalize: When Non-identifiability is a Problem

Yaniv Yacoby, Weiwei Pan, Finale Doshi-Velez

Abstract

Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this work, we first show that BNN+LV suffers from a serious form of non-identifiability: explanatory power can be transferred between the model parameters and latent variables while fitting the data equally well. We demonstrate that as a result, in the limit of infinite data, the posterior mode over the network weights and latent variables is asymptotically biased away from the ground-truth. Due to this asymptotic bias, traditional inference methods may in practice yield parameters that generalize poorly and misestimate uncertainty. Next, we develop a novel inference procedure that explicitly mitigates the effects of likelihood non-identifiability during training and yields high-quality predictions as well as uncertainty estimates. We demonstrate that our inference method improves upon benchmark methods across a range of synthetic and real data-sets.

Introduction

In many high-stakes machine learning applications—such as active learning or safe reinforcement learning—it is insufficient to merely predict an outcome; one must also understand the source of uncertainty. Uncertainty is generally categorized into two types: epistemic (uncertainty in the model weights due to a lack of data) and aleatoric (inherent randomness in the environment).

Bayesian Neural Networks with Latent Variables (BNN+LVs) are designed to separate these components by utilizing weights ($W$) for epistemic uncertainty and a latent input variable ($z$) for aleatoric uncertainty. This architecture allows the model to handle "heteroscedastic" noise—noise that varies depending on the input $x$.

The Non-identifiability Problem

The primary challenge identified in this research is non-identifiability. Because both the weights $W$ and the latent variables $z$ are unknown, the model can "trade" explanatory power between them. For example, the latent variable $z$ might inadvertently learn a mapping of the input $x$, allowing the weights $W$ to fit a simpler, incorrect function that appears accurate on training data but fails to generalize to new data.

Graphical model of a BNN+LV.

The authors provide a formal proof that this is not merely a training quirk; even in the limit of infinite data, the joint posterior mode of a BNN+LV is asymptotically biased away from the ground truth. This "memorization" pathology explains why traditional Maximum A Posteriori (MAP) estimates and standard mean-field Variational Inference (VI) often yield poor results for these models.

Noise-Constrained Approximate Inference (NCAI)

To mitigate the effects of non-identifiability, the authors introduce Noise-Constrained Approximate Inference (NCAI). This method forces the model to respect its own structural assumptions during training by adding two explicit constraints to the standard Variational Inference objective:

  1. Independence Constraint: This forces the inferred latent variables $z$ to be independent of the inputs $x$, preventing $z$ from memorizing $x$.
  2. Distribution Constraint: This ensures the aggregate distribution of the inferred $z$ values matches the prior distribution (typically a standard Gaussian).

Because direct calculation of these constraints is mathematically intractable, the authors employ differentiable proxies. For independence, they penalize the correlation between $x$ and $z$. For the distribution match, they utilize the Henze-Zirkler test, a statistical test for normality, as a penalty term. These are solved as a constrained optimization problem using a Lagrangian formulation.

Experimental Results

The researchers evaluated NCAI against standard benchmarks using both synthetic and real-world datasets, including Heavy-Tail, Bimodal, and various UCI regression tasks like Yacht Hydrodynamics and Wine Quality.

Key findings include:

  • Superior Generalization: NCAI consistently achieved higher test log-likelihoods and lower Root Mean Squared Error (RMSE) compared to standard unconstrained BNN+LVs.
  • Accurate Uncertainty Decomposition: NCAI produced uncertainty estimates that closely matched "gold standard" Hamiltonian Monte Carlo (HMC) results, correctly distinguishing between aleatoric and epistemic uncertainty where standard VI failed.
  • Structural Regularization: The results confirm that "structural" regularization—forcing the model to obey its generative assumptions—is more effective for latent variable models than standard weight decay or dropout.

Conclusion and Future Work

This work highlights a fundamental pathology in BNN+LVs where latent variables act as high-capacity memory for the training set. While NCAI effectively addresses this through constrained optimization, the authors note that the choice of proxies (like Pearson correlation) is heuristic and may not capture all non-linear dependencies. Future research may explore more robust independence measures and investigate whether similar non-identifiability issues affect other latent variable models, such as Variational Autoencoders (VAEs).

Summary updated

Abstract

Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this work, we first show that BNN+LV suffers from a serious form of non-identifiability: explanatory power can be transferred between the model parameters and latent variables while fitting the data equally well. We demonstrate that as a result, in the limit of infinite data, the posterior mode over the network weights and latent variables is asymptotically biased away from the ground-truth. Due to this asymptotic bias, traditional inference methods may in practice yield parameters that generalize poorly and misestimate uncertainty. Next, we develop a novel inference procedure that explicitly mitigates the effects of likelihood non-identifiability during training and yields high-quality predictions as well as uncertainty estimates. We demonstrate that our inference method improves upon benchmark methods across a range of synthetic and real data-sets.
BibTeX
@misc{yacoby2019learning,
  author = {Yaniv Yacoby and Weiwei Pan and Finale Doshi-Velez},
  title = {{Learning Deep Bayesian Latent Variable Regression Models that Generalize: When Non-identifiability is a Problem}},
  year = {2019},
  eprint = {1911.00569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.00569},
}
Figure