Abstract: Amortized inference, particularly in Variational Auto-encoders (VAEs), relies on an inference network (encoder) to approximate the posterior distribution of latent variables. A critical yet often overlooked design choice is the coarse-grained structure of this network—specifically, the dependency graph between variables. We introduce NaMI (Natural Minimal I-map generator), an algorithm that automatically constructs optimal inference structures by simulating exact variable elimination. NaMI ensures that the resulting inverse model is "faithful" (capturing all necessary dependencies) and "minimal" (avoiding redundant edges that introduce noise). Our evaluations across MNIST, Gaussian Belief Networks, and Gaussian Mixture Models show that NaMI-generated structures consistently yield tighter variational bounds and faster convergence than standard heuristics or fully connected baselines.
Introduction: The Structure of Amortized Inference
In amortized inference, we train a neural network to predict latent variable distributions given observed data. While much research focuses on the "fine-grained" architecture (layer depth, widths), the "coarse-grained" structure—the factorization of the approximate posterior—is often chosen via simple heuristics. Common approaches include reversing the arrows of the generative model or using a fully connected graph.
However, these heuristics frequently fail to capture the "explaining away" effect, where independent causes become dependent upon observing their common effect. Failing to account for these dependencies leads to "unfaithful" models that cannot accurately represent the true posterior. Conversely, fully connected models, while expressive, ignore the problem's inherent structure, leading to noisy gradients and slower training.
Methods: The NaMI Algorithm
To address these issues, we propose NaMI (Natural Minimal I-map generator), a principled algorithm for graphical model inversion. NaMI works by simulating a classic exact inference algorithm called variable elimination. It tracks how dependencies "flow" through the graph as variables are marginalized out.
The algorithm ensures the resulting inverse is both faithful and minimal. A model is faithful if it encodes no independence assertions that are absent from the original generative model. It is minimal if the removal of any single edge would violate this faithfulness. By maintaining minimality, NaMI acts as a structural regularizer, preventing the introduction of unnecessary edges that would otherwise act as sources of random noise during stochastic gradient descent.
NaMI operates with a computational complexity of $O(nc)$, where $n$ is the number of variables, making it an efficient pre-processing step. It supports both topological and reverse-topological orderings, ensuring the sampling order of the inference network feels "natural" relative to the original generative process.
Experimental Evaluation
The authors evaluated NaMI across three distinct scenarios:
- MNIST: Used for training "Relaxed Bernoulli VAEs" (a continuous relaxation of discrete latent variable models).
- Synthetic Binary-tree Gaussian BNs: A controlled environment where the exact posterior is known analytically, allowing for precise measurement of error.
- Synthetic Bayesian Gaussian Mixture Models (GMMs): A classic clustering task used to test how the algorithm handles complex dependencies between cluster assignments and global parameters.
Key Results
- Tighter Variational Bounds: In the Relaxed Bernoulli VAE experiments on MNIST, the faithful inverse consistently achieved a lower negative ELBO and better AIS (Annealed Importance Sampling) estimates than the standard mean-field (heuristic) approach across all parameter scales.
- Faster Convergence: For Gaussian Mixture Models, the NaMI-generated structure converged significantly faster and to a better final solution than a fully connected baseline.
- Stability: The authors noted that while fully connected or heuristic models became unstable when network capacity was reduced, the minimally faithful inverses remained stable and effective.
Conclusion
NaMI demonstrates that the coarse-grained structure of an inference network is a prerequisite for effective amortized inference. By automatically generating faithful and minimal inverse models, NaMI provides a robust foundation for neural network training, leading to more accurate and efficient posterior approximations. While the user still must choose the distribution family for each factor, NaMI ensures the underlying dependency structure is optimal.
Summary updated