Abstract
We propose a new and general approach to learn deep generative models. Our approach is based on a new observation that the importance weighted autoencoders (IWAE, Burda et al. (2015)) can be understood as a specific form of the Expectation-Maximization (EM) algorithm. By reinterpreting IWAE through this lens, we introduce Importance Weighted EM (IWEM), which utilizes an optimal proposal distribution and an annealing strategy to significantly outperform standard IWAE, particularly in challenging missing-data scenarios.
Introduction
Deep generative models, such as Variational Autoencoders (VAEs), have revolutionized how we model complex data distributions. VAEs typically maximize the Evidence Lower Bound (ELBO), which can be a loose approximation of the true log-likelihood. The Importance Weighted Autoencoder (IWAE) was introduced to tighten this bound using multiple samples. However, the theoretical underpinnings of IWAE's objective were previously viewed as a heuristic. This paper, presented at AISTATS 2020, provides a fresh perspective by casting IWAE as an Expectation-Maximization (EM) algorithm.
Theoretical Framework: IWAE as EM
The core theoretical contribution is the proof that the gradient of the IWAE objective is equivalent to the gradient of an EM objective using self-normalized importance sampling (snIS). This realization shifts the focus from merely "finding a tighter bound" to "directly performing Maximum Likelihood Estimation (MLE)" via an approximate EM framework.
Furthermore, the authors provide a theoretical justification for IWAE’s divergence measure, showing that as the number of samples ($K$) goes to infinity, the objective converges to the Chi-squared ($\chi^2$) distance between the proposal and the true posterior.
The IWEM Algorithm
Building on the EM interpretation, the authors propose Importance Weighted EM (IWEM), which introduces two key enhancements to the standard IWAE framework:
Optimal P-step
In the EM framework, the "Proposal step" (P-step) updates the variational distribution. The authors show that the standard IWAE update is not variance-optimal. They propose a new objective for the P-step that minimizes the $\chi^2$ distance between the proposal and the true posterior, leading to more stable and efficient learning.
Annealing Strategy
To manage the variance-bias trade-off, IWEM employs an annealing strategy. Early in training, the algorithm relies on the low-variance ELBO. As training progresses, a controller ($\alpha$) shifts the objective toward the higher-variance but unbiased EM objective. This "warm-up" ensures stability in the early stages while achieving high accuracy in the later stages.
Handling Missing Data: miss-IWEM
The EM framework is naturally suited for missing data. By treating missing values as additional latent variables to be sampled, the authors extend their method to miss-IWEM. This approach is particularly effective for "patch-wise removal" tasks in images, where entire coherent chunks of an image are missing, requiring the model to understand global structure for reconstruction.
Experimental Results
The authors evaluated IWEM on standard image benchmarks (MNIST, Omniglot, Caltech 101) and UCI tabular datasets.
Generative Performance
IWEM consistently achieved higher test log-likelihoods than IWAE and other state-of-the-art methods like DReG. The annealing strategy and optimal P-step proved crucial for these gains, allowing the model to converge to better solutions.
Missing Data and Imputation
In missing data scenarios, miss-IWEM demonstrated significant superiority. On the biMNIST dataset with 6 patches removed, miss-IWEM achieved a log-likelihood of -95.18, compared to -102.26 for missIWAE. Qualitatively, the model produces coherent reconstructions of missing image parts, as shown below.
On tabular data (UCI), miss-IWEM significantly reduced Mean-Squared Error (MSE) for feature imputation. For example, on the 'Bank' dataset, MSE dropped from 0.598 (missIWAE) to 0.465 (miss-IWEM).
Discussion and Limitations
While IWEM offers significant improvements, the authors noted that the annealing strategy is not always perfectly compatible with the optimal P-step in fully observed cases, sometimes introducing a bias that can hinder finding the absolute best solution. Additionally, the computational footprint is higher than a simple VAE due to the need for multiple samples ($K$ and $J$).
Conclusion
By reinterpreting IWAE as an EM algorithm, this work provides both a stronger theoretical foundation and a more powerful algorithmic framework for deep generative modeling. IWEM's ability to handle missing data and its improved generative performance make it a robust alternative to standard VAE and IWAE approaches. Future work may explore its application to disentangled representation learning.
Summary updated