On casting importance weighted autoencoder to an EM algorithm to learn deep generative models

Dongha Kim, Jaesung Hwang, Yongdai KimAISTATS 2020 · OpenReview

Abstract

We propose a new and general approach to learn deep generative models. Our approach is based on a new observation that the importance weighted autoencoders (IWAE, Burda et al. (2015)) can be understood as a specific form of the Expectation-Maximization (EM) algorithm. By reinterpreting IWAE through this lens, we introduce Importance Weighted EM (IWEM), which utilizes an optimal proposal distribution and an annealing strategy to significantly outperform standard IWAE, particularly in challenging missing-data scenarios.

Introduction

Deep generative models, such as Variational Autoencoders (VAEs), have revolutionized how we model complex data distributions. VAEs typically maximize the Evidence Lower Bound (ELBO), which can be a loose approximation of the true log-likelihood. The Importance Weighted Autoencoder (IWAE) was introduced to tighten this bound using multiple samples. However, the theoretical underpinnings of IWAE's objective were previously viewed as a heuristic. This paper, presented at AISTATS 2020, provides a fresh perspective by casting IWAE as an Expectation-Maximization (EM) algorithm.

Theoretical Framework: IWAE as EM

The core theoretical contribution is the proof that the gradient of the IWAE objective is equivalent to the gradient of an EM objective using self-normalized importance sampling (snIS). This realization shifts the focus from merely "finding a tighter bound" to "directly performing Maximum Likelihood Estimation (MLE)" via an approximate EM framework.

Furthermore, the authors provide a theoretical justification for IWAE’s divergence measure, showing that as the number of samples ($K$) goes to infinity, the objective converges to the Chi-squared ($\chi^2$) distance between the proposal and the true posterior.

The IWEM Algorithm

Building on the EM interpretation, the authors propose Importance Weighted EM (IWEM), which introduces two key enhancements to the standard IWAE framework:

Optimal P-step

In the EM framework, the "Proposal step" (P-step) updates the variational distribution. The authors show that the standard IWAE update is not variance-optimal. They propose a new objective for the P-step that minimizes the $\chi^2$ distance between the proposal and the true posterior, leading to more stable and efficient learning.

Comparison of training performance with and without the optimal P-step. The optimal P-step consistently yields lower error metrics across epochs.

Annealing Strategy

To manage the variance-bias trade-off, IWEM employs an annealing strategy. Early in training, the algorithm relies on the low-variance ELBO. As training progresses, a controller ($\alpha$) shifts the objective toward the higher-variance but unbiased EM objective. This "warm-up" ensures stability in the early stages while achieving high accuracy in the later stages.

Handling Missing Data: miss-IWEM

The EM framework is naturally suited for missing data. By treating missing values as additional latent variables to be sampled, the authors extend their method to miss-IWEM. This approach is particularly effective for "patch-wise removal" tasks in images, where entire coherent chunks of an image are missing, requiring the model to understand global structure for reconstruction.

Experimental Results

The authors evaluated IWEM on standard image benchmarks (MNIST, Omniglot, Caltech 101) and UCI tabular datasets.

Generative Performance

IWEM consistently achieved higher test log-likelihoods than IWAE and other state-of-the-art methods like DReG. The annealing strategy and optimal P-step proved crucial for these gains, allowing the model to converge to better solutions.

Missing Data and Imputation

In missing data scenarios, miss-IWEM demonstrated significant superiority. On the biMNIST dataset with 6 patches removed, miss-IWEM achieved a log-likelihood of -95.18, compared to -102.26 for missIWAE. Qualitatively, the model produces coherent reconstructions of missing image parts, as shown below.

Reconstruction of missing patches in MNIST digits using the proposed framework. The columns represent the masked input (left), the IWEM reconstruction (middle), and the ground truth (right).

On tabular data (UCI), miss-IWEM significantly reduced Mean-Squared Error (MSE) for feature imputation. For example, on the 'Bank' dataset, MSE dropped from 0.598 (missIWAE) to 0.465 (miss-IWEM).

Discussion and Limitations

While IWEM offers significant improvements, the authors noted that the annealing strategy is not always perfectly compatible with the optimal P-step in fully observed cases, sometimes introducing a bias that can hinder finding the absolute best solution. Additionally, the computational footprint is higher than a simple VAE due to the need for multiple samples ($K$ and $J$).

Conclusion

By reinterpreting IWAE as an EM algorithm, this work provides both a stronger theoretical foundation and a more powerful algorithmic framework for deep generative modeling. IWEM's ability to handle missing data and its improved generative performance make it a robust alternative to standard VAE and IWAE approaches. Future work may explore its application to disentangled representation learning.

Summary updated

Abstract

We propose a new and general approach to learn deep generative models. Our approach is based on a new observation that the importance weighted autoencoders (IWAE, Burda et al. (2015)) can be unders...
BibTeX
@inproceedings{kim2020casting,
  author = {Dongha Kim and Jaesung Hwang and Yongdai Kim},
  title = {{On casting importance weighted autoencoder to an EM algorithm to learn deep generative models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=WvPToDRu9L6},
}
Figure