Distribution Matching in Variational Inference

Mihaela Rosca, Balaji Lakshminarayanan, Shakir MohamedarXiv 2018 · arXiv

Abstract

Variational Autoencoders (VAEs) are a cornerstone of generative modeling, offering stable training and a principled framework for latent variable inference. However, a persistent challenge remains: the "marginal matching" problem, where the aggregate posterior distribution fails to align with the prior, resulting in "holes" in the latent space and poor sample quality. This paper explores distribution matching techniques, specifically VAE-GAN hybrids utilizing the "density ratio trick," to bridge this gap. Through extensive benchmarking on ColorMNIST, CelebA, and CIFAR-10, we find that while adversarial matching improves sample sharpness, it does not fundamentally resolve the limitations of the VAE objective and introduces new complexities in scalability and evaluation.

Introduction: The Gap in Generative Modeling

Generative modeling is currently dominated by two primary paradigms: Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). VAEs are favored for their stability and the clear mathematical lower bound (ELBO) they provide for training. GANs, conversely, are known for producing high-fidelity images but are notoriously difficult to train and evaluate.

This research addresses a specific failure mode in VAEs: the discrepancy between conditional and marginal distribution matching. While VAEs are designed to match conditional distributions—ensuring a specific latent code $z$ can reconstruct a specific image $x$—they often fail to match the marginal distribution. This means the aggregate of all latent codes $q(z)$ does not align with the prior $p(z)$ (typically a simple Gaussian). When this alignment fails, "holes" appear in the latent space. Sampling from these regions results in "bad" samples that bear little resemblance to the training data, even if the model's reconstruction capabilities are excellent.

Methods: Bridging the Gap with Distribution Matching

The researchers investigate "Distribution Matching" as a mechanism to bridge the gap between VAEs and GANs. They contrast standard VAEs with several hybrid models that employ the "Density Ratio Trick."

  1. The Density Ratio Trick: Instead of relying on fixed mathematical formulas like KL divergence to compare distributions, a GAN-style discriminator is trained to distinguish between them. The discriminator's output provides an estimate of the ratio between the two distributions, which can then be used as a training signal.
  2. Adversarial Autoencoders (AAE): These models apply the density ratio trick within the latent space, forcing the aggregate posterior $q(z)$ to match the prior $p(z)$.
  3. Variational GAN Hybrids (VGH & VGH++): Introduced by the authors, these models apply adversarial matching in the visible (image) space. VGH++ specifically utilizes a discriminator to evaluate both reconstructions and random samples, encouraging the model to cover the entire data manifold.
  4. VEEGAN: This approach focuses on joint-distribution matching, attempting to align the pair $(x, z)$ across both the encoder and decoder networks.

Key Insights: Complexity vs. Objective

A common hypothesis is that VAEs fail because their posterior approximations are too simplistic. However, the authors demonstrate that increasing model complexity is not a panacea. Even when using Normalizing Flows (RNVP)—highly flexible distribution approximators—the marginal matching problem persists.

Figure 6: Comparison of Mutual Information and Marginal KL for VAE-Gauss and VAE-RNVP on the CelebA dataset. The persistent Marginal KL gap indicates that increased posterior complexity does not resolve the distribution matching issue.

This suggests that the issue is inherent to the VAE objective itself rather than the capacity of the networks. Furthermore, the "Classifier Trap" reveals a limitation of the density ratio trick: in high-dimensional spaces, discriminators consistently underestimate the true divergence between distributions. This makes them unreliable for model evaluation, even if they remain useful for training.

The study also highlights a difference in representation learning. VAEs tend to learn "sparse" representations, often ignoring many latent dimensions. In contrast, adversarial models like AAEs force the use of the entire latent space, leading to "dense" representations. This property may make AAEs more suitable for downstream tasks such as semi-supervised learning.

Key Results and Qualitative Analysis

The models were evaluated on ColorMNIST, CelebA, and CIFAR-10. CIFAR-10, with its diverse set of objects, proved significantly more challenging than the face-centric CelebA dataset.

Figure 1: Generated samples on the CelebA dataset. While VAE-based models capture the general structure of faces, they often suffer from characteristic blurriness compared to pure GAN approaches.
  • Qualitative Improvement: VAE-GAN hybrids, particularly VGH++, produce noticeably sharper and more realistic samples than standard VAEs. This is especially evident on complex datasets like CIFAR-10.
  • The GAN Ceiling: Despite the increased complexity of hybrid models, they did not surpass pure GANs (such as WGAN-GP) in terms of raw image quality or Inception Scores.
  • Evaluation Challenges: The reliance on discriminators in hybrid models removes the ability to calculate a reliable ELBO. Consequently, there is no single metric to track during training to ensure convergence, leading to potential oscillations.

Limitations and Conclusion

The authors conclude that VAE-GAN hybrids are not a "silver bullet" for generative modeling. They identify three primary bottlenecks:

  1. Scalability: Adversarial components struggle as the dimensionality of the latent space increases. While standard VAEs scale easily to thousands of latent dimensions, AAEs often fail in these regimes.
  2. Hyperparameter Sensitivity: Hybrids are significantly more difficult to tune than standard VAEs, requiring careful balancing of adversarial and variational losses.
  3. Future Directions: The paper suggests that future research should move beyond binary classifiers for distribution matching in high-dimensional spaces. Finding more robust ways to align distributions remains the primary hurdle for scaling these hybrid models to more complex data.

Summary updated

Abstract

With the increasingly widespread deployment of generative models, there is a mounting need for a deeper understanding of their behaviors and limitations. In this paper, we expose the limitations of Variational Autoencoders (VAEs), which consistently fail to learn marginal distributions in both latent and visible spaces. We show this to be a consequence of learning by matching conditional distributions, and the limitations of explicit model and posterior distributions. It is popular to consider Generative Adversarial Networks (GANs) as a means of overcoming these limitations, leading to hybrids of VAEs and GANs. We perform a large-scale evaluation of several VAE-GAN hybrids and analyze the implications of class probability estimation for learning distributions. While promising, we conclude that at present, VAE-GAN hybrids have limited applicability: they are harder to scale, evaluate, and use for inference compared to VAEs; and they do not improve over the generation quality of GANs.
BibTeX
@misc{rosca2018distribution,
  author = {Mihaela Rosca and Balaji Lakshminarayanan and Shakir Mohamed},
  title = {{Distribution Matching in Variational Inference}},
  year = {2018},
  eprint = {1802.06847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.06847},
}
Figure