Abstract
Inference for Variational Autoencoders (VAEs) consists of learning two models: (1) a generative model, which transforms a simple distribution over a latent space into the distribution over observed data, and (2) an inference model, which approximates the posterior of the latent codes given data. The two components are learned jointly via a lower bound to the generative model's log marginal likelihood. In early phases of joint training, the inference model poorly approximates the latent code posteriors. Recent work showed that this leads optimization to get stuck in local optima, negatively impacting the learned generative model. As such, recent work suggests ensuring a high-quality inference model via iterative training: maximizing the objective function relative to the inference model before every update to the generative model. Unfortunately, iterative training is inefficient, requiring heuristic criteria for reverting from iterative to joint training for speed. Here, we suggest an inference method that trains the generative and inference models independently. It approximates the posterior of the true model a priori; fixing this posterior approximation, we then maximize the lower bound relative to only the generative model. By conventional wisdom, this approach should rely on the true prior and likelihood of the true model to approximate its posterior (which are unknown). However, we show that we can compute a deterministic, model-agnostic posterior approximation (MAPA) of the true model's posterior. We then use MAPA to develop a proof-of-concept inference method. We present preliminary results on low-dimensional synthetic data that (1) MAPA captures the trend of the true posterior, and (2) our MAPA-based inference performs better density estimation with less computation than baselines. Lastly, we present a roadmap for scaling the MAPA-based inference method to high-dimensional data.
Introduction
Training a Variational Autoencoder (VAE) involves a "chicken-and-egg" problem: learning a generative model (decoder) requires a good inference model (encoder), and vice versa. Standard joint optimization often leads to suboptimal solutions because the inference model is poor in early training stages. While iterative training—updating the encoder multiple times per decoder update—can mitigate this, it is computationally expensive and relies on heuristics. This work introduces Model-Agnostic Posterior Approximation (MAPA), a method to decouple these components by estimating the latent posterior based solely on the proximity of data points in the observation space.
Methodology: Model-Agnostic Posterior Approximation (MAPA)
MAPA shifts the perspective to "Empiricalized" Models, using an empirical distribution of latent codes. This allows treating inference as estimating the probability of a latent code index rather than a specific location in latent space.
The core of MAPA is its model-agnosticism. It approximates the posterior $p(z|x)$ using a proximity kernel $\kappa(x_n | x_i)$ (e.g., a Gaussian RBF kernel) between observations. The intuition is that if two data points are close in the observation space, their latent codes should have a high probability of being interchangeable under the posterior. This approximation is deterministic and can be pre-computed once per dataset.
A key advantage of MAPA is its robustness to model non-identifiability. Since many decoders can produce the same data distribution, MAPA's reliance on data geometry ensures it captures the posterior trend regardless of the specific decoder variant being learned. Furthermore, MAPA enables a new importance-weighted lower bound that uses the fixed MAPA distribution as a proposal, avoiding the need for specialized gradient estimators for inference parameters.
Experimental Evaluation
The method was evaluated on five synthetic 1D latent space datasets (including "Spiral-Dots" and "Absolute-Value") with $N=5000$ points each. These datasets highlight specific VAE pathologies where standard mean-field Gaussian VAEs typically struggle.
Key Results
MAPA demonstrates superior performance in both accuracy and efficiency compared to standard VAE and Importance Weighted Autoencoder (IWAE) baselines.
- Better Density Estimation: Across nearly all test cases, MAPA achieved a lower test KL divergence, indicating a better match to the true data distribution than both standard VAEs and IWAEs.
- Computational Efficiency: MAPA significantly reduces computational overhead. A MAPA model with $S=200$ importance samples requires roughly the same number of neural network forward passes as an IWAE with only $S=50$ to $100$ samples. This efficiency stems from the ability to reuse decoder forward passes across different points in a batch.
Discussion and Future Work
While MAPA shows promise on low-dimensional data, scaling to high-dimensional spaces like images presents challenges. The "curse of dimensionality" can make simple distance kernels less meaningful, and the current bound neglects latent code entropy. Additionally, the $O(N^2)$ cost of computing the proximity kernel for large datasets necessitates the use of sketching algorithms or sparse Kernel Density Estimation (KDE). Future research will also explore more general kernels for complex data types like time-series and refine noise model assumptions.
Summary updated