Revisiting Reweighted Wake-Sleep

Tuan Anh Le, Adam R. Kosiorek, N. Siddharth, Yee Whye Teh, Frank Wood

TL;DR

This paper revisits the Reweighted Wake-Sleep (RWS) algorithm for learning complex generative models with stochastic control flow. They demonstrate that RWS outperforms existing state-of-the-art methods, particularly in scenarios where other approaches struggle, offering a more robust and often preferable alternative for learning these challenging models.

What Problem Does This Paper Address?

Generative models with stochastic control flow (SCFMs) are powerful tools for modeling complex data by incorporating branching decisions based on discrete random variables. However, learning these models using standard gradient-based methods is notoriously difficult. Many existing approaches rely on continuous relaxations of discrete variables, which become computationally intractable for SCFMs due to the need to evaluate exponentially many branching paths. While alternatives like REINFORCE with control variates exist, they often suffer from high variance or can even degrade performance as more computational resources (particles) are used. This paper aims to demonstrate that a revisited RWS algorithm offers a more effective and stable solution for learning SCFMs.

Methods

The paper revisits and extensively evaluates the Reweighted Wake-Sleep (RWS) algorithm. RWS is an extension of the wake-sleep algorithm that jointly learns a generative model and an inference network. Unlike methods that optimize a single objective, RWS alternates between optimizing individual objectives for the generative model (wake-phase) and the inference network (sleep-phase or wake-phase). The key innovation lies in how RWS updates the inference network by directly minimizing the KL divergence between the model's posterior and the inference network's approximation, which is shown to be more stable and effective than objectives used in methods like Importance Weighted Autoencoders (IWAE). The authors compare RWS against IWAE combined with control variates and continuous relaxations.

Datasets

The paper evaluates the proposed methods on three distinct datasets/benchmarks:

  • Probabilistic Context-Free Grammar (PCFG): Used to model sentences, this benchmark is notable because continuous relaxations are inapplicable due to potentially infinite recursion.
  • Attend, Infer, Repeat (AIR): A complex visual generative model for perceiving and localizing objects, which involves both discrete and continuous latent variables and a sequential inference process.
  • Gaussian Mixture Model (GMM): A simplified, pedagogical example used to isolate and understand specific aspects of the algorithms' behavior, particularly concerning branching and posterior approximation.

Key Insights and Ideas

  • RWS is a robust alternative for SCFMs: The paper strongly advocates for RWS, showing it consistently outperforms or matches state-of-the-art methods, especially in challenging SCFM scenarios.
  • The curse of particles for IWAE: Increasing the number of particles (K) in IWAE can paradoxically harm the learning of the inference network, leading to degraded generative model performance. RWS, in contrast, benefits from more particles.
  • Directly minimizing KL divergence is key: RWS's approach of directly targeting the KL divergence between the true posterior and the approximate posterior for inference network updates is a crucial advantage.
  • Branching is the core challenge: The paper highlights how methods relying on continuous relaxations fail when faced with explicit branching in SCFMs, necessitating alternative approaches like RWS.
  • A fix for branch-pruning: The authors identify a "branch-pruning" failure mode in RWS (and other methods) under specific initializations and propose a simple extension (δ-ww) to mitigate it.

Key Results

  • PCFG: RWS outperforms IWAE with control variates in both learning and inference amortization, particularly where continuous relaxations are not applicable.
  • AIR: RWS shows monotonic improvement with increasing particles, while VIMCO's performance degrades for the inference network. RWS also yields significantly lower variance and better inference networks.
  • GMM: RWS and its variants (ws, ww) demonstrate better inference network learning with increased particle budgets, unlike IWAE methods. The paper also analyzes the impact of initialization and proposes δ-ww to address branch-pruning, showing it can outperform other methods in low-particle regimes.

Limitations and Future Work

  • Bias in RWS gradient estimators: The paper acknowledges that both sleep-phase and wake-phase gradient estimators in RWS are biased. While theoretically manageable, this bias can be significant in practice, especially when the model distribution is far from the true data distribution.
  • Architecture design for control variates: For methods like REBAR/RELAX, designing architectures that are highly correlated with the high-variance term and stable to train remains a challenge.
  • Branch-pruning mitigation: While δ-ww is proposed as a solution for branch-pruning, further investigation into its theoretical underpinnings and broader applicability could be beneficial.
  • Generalizability of RWS: The paper suggests that the choice between RWS variants (ws vs. ww) depends on which gradient bias dominates. Further empirical studies across a wider range of SCFMs could solidify these findings.

Summary updated

BibTeX
@inproceedings{le2019revisiting,
  author = {Tuan Anh Le and Adam R. Kosiorek and N. Siddharth and Yee Whye Teh and Frank Wood},
  title = {{Revisiting Reweighted Wake-Sleep}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=yFaEbwO7mv6},
}
Figure