Faithful Model Inversion Substantially Improves Auto-encoding Variational Inference

Stefan Webb, Adam Golinski, Robert Zinkov, N. Siddharth, Yee Whye Teh, Frank D. Wood

Abstract: Amortized inference, particularly in Variational Auto-encoders (VAEs), relies on an inference network (encoder) to approximate the posterior distribution of latent variables. A critical yet often overlooked design choice is the coarse-grained structure of this network—specifically, the dependency graph between variables. We introduce NaMI (Natural Minimal I-map generator), an algorithm that automatically constructs optimal inference structures by simulating exact variable elimination. NaMI ensures that the resulting inverse model is "faithful" (capturing all necessary dependencies) and "minimal" (avoiding redundant edges that introduce noise). Our evaluations across MNIST, Gaussian Belief Networks, and Gaussian Mixture Models show that NaMI-generated structures consistently yield tighter variational bounds and faster convergence than standard heuristics or fully connected baselines.

Introduction: The Structure of Amortized Inference

In amortized inference, we train a neural network to predict latent variable distributions given observed data. While much research focuses on the "fine-grained" architecture (layer depth, widths), the "coarse-grained" structure—the factorization of the approximate posterior—is often chosen via simple heuristics. Common approaches include reversing the arrows of the generative model or using a fully connected graph.

However, these heuristics frequently fail to capture the "explaining away" effect, where independent causes become dependent upon observing their common effect. Failing to account for these dependencies leads to "unfaithful" models that cannot accurately represent the true posterior. Conversely, fully connected models, while expressive, ignore the problem's inherent structure, leading to noisy gradients and slower training.

Comparison of generative (a), naive inverse (b), and faithful inverse (c) structures.

Methods: The NaMI Algorithm

To address these issues, we propose NaMI (Natural Minimal I-map generator), a principled algorithm for graphical model inversion. NaMI works by simulating a classic exact inference algorithm called variable elimination. It tracks how dependencies "flow" through the graph as variables are marginalized out.

The algorithm ensures the resulting inverse is both faithful and minimal. A model is faithful if it encodes no independence assertions that are absent from the original generative model. It is minimal if the removal of any single edge would violate this faithfulness. By maintaining minimality, NaMI acts as a structural regularizer, preventing the introduction of unnecessary edges that would otherwise act as sources of random noise during stochastic gradient descent.

Generative model G and candidate inverses H1, H2, H3. H1 is faithful and minimal; H2 is unfaithful; H3 is faithful but not minimal.

NaMI operates with a computational complexity of $O(nc)$, where $n$ is the number of variables, making it an efficient pre-processing step. It supports both topological and reverse-topological orderings, ensuring the sampling order of the inference network feels "natural" relative to the original generative process.

Experimental Evaluation

The authors evaluated NaMI across three distinct scenarios:

  • MNIST: Used for training "Relaxed Bernoulli VAEs" (a continuous relaxation of discrete latent variable models).
  • Synthetic Binary-tree Gaussian BNs: A controlled environment where the exact posterior is known analytically, allowing for precise measurement of error.
  • Synthetic Bayesian Gaussian Mixture Models (GMMs): A classic clustering task used to test how the algorithm handles complex dependencies between cluster assignments and global parameters.

Key Results

  • Tighter Variational Bounds: In the Relaxed Bernoulli VAE experiments on MNIST, the faithful inverse consistently achieved a lower negative ELBO and better AIS (Annealed Importance Sampling) estimates than the standard mean-field (heuristic) approach across all parameter scales.
  • Faster Convergence: For Gaussian Mixture Models, the NaMI-generated structure converged significantly faster and to a better final solution than a fully connected baseline.
  • Stability: The authors noted that while fully connected or heuristic models became unstable when network capacity was reduced, the minimally faithful inverses remained stable and effective.

Conclusion

NaMI demonstrates that the coarse-grained structure of an inference network is a prerequisite for effective amortized inference. By automatically generating faithful and minimal inverse models, NaMI provides a robust foundation for neural network training, leading to more accurate and efficient posterior approximations. While the user still must choose the distribution family for each factor, NaMI ensures the underlying dependency structure is optimal.

Summary updated

Abstract

In learning deep generative models, the encoder for variational inference is typically formed in an ad hoc manner with a structure and parametrization analogous to the forward model. Our chief insight is that this results in coarse approximations to the posterior, and that the d-separation properties of the BN structure of the forward model should be used, in a principled way, to produce ones that are faithful to the posterior, for which we introduce the novel Compact Minimal I-map (CoMI) algorithm. Applying our method to common models reveals that standard encoder design choices lack many important edges, and through experiments we demonstrate that modelling these edges is important for optimal learning. We show how using a faithful encoder is crucial when modelling with continuous relaxations of categorical distributions.
BibTeX
@misc{webb2017faithful,
  author = {Stefan Webb and Adam Golinski and Robert Zinkov and N. Siddharth and Yee Whye Teh and Frank D. Wood},
  title = {{Faithful Model Inversion Substantially Improves Auto-encoding Variational Inference}},
  year = {2017},
  url = {https://openreview.net/forum?id=lNnjnv2hYS},
}
Figure