Learning Deep-Latent Hierarchies by Stacking Wasserstein Autoencoders

Benoit Gaujac, Ilya Feige, David BarberarXiv 2020 · OpenReview

TL;DR

This paper introduces STACKEDWAE, a novel method for training deep generative models with hierarchical latent spaces using Optimal Transport. By recursively stacking Wasserstein Autoencoders (WAEs), STACKEDWAE effectively leverages deep latent hierarchies, overcoming the "latent variable collapse" issue common in VAEs and leading to improved sample generation and more interpretable latent representations.

What Problem Does This Paper Address?

Deep generative models with hierarchical latent variables hold great promise for capturing complex data structures. However, training these models, particularly those based on Variational Autoencoders (VAEs), is notoriously difficult. A key challenge is ensuring that the model effectively utilizes its deep latent hierarchy, rather than collapsing to simpler, shallower representations. Existing approaches often require complex, custom-designed inference networks and optimization schemes. This paper aims to provide a more principled and generalizable way to train deep-latent-variable models by leveraging the theoretical advantages of Optimal Transport (OT).

Methods

The core contribution is the STACKEDWAE objective function. This method builds upon the Wasserstein Autoencoder (WAE) framework, which uses Optimal Transport to define a non-likelihood-based training objective. Instead of a single WAE, STACKEDWAE recursively "stacks" WAEs. At each level of the hierarchy, a WAE is trained to map data to a latent representation, and its objective is regularized by the Wasserstein distance to the distribution of the next deeper latent layer. This recursive stacking, combined with a specific formulation of the inference network's latent hierarchy, allows the model to learn a smooth and usable representation across all latent layers. The authors derive this objective by extending the original WAE formulation and show that it naturally encourages the use of the entire latent hierarchy.

Datasets

The authors evaluated their method on the following datasets:

  • MNIST: A standard dataset of handwritten digits.
  • Street View House Numbers (SVHN): A dataset of house numbers.
  • CelebA: A dataset of celebrity faces.

Key Insights and Ideas

  • Recursive WAE stacking is key: The paper demonstrates that by recursively applying the WAE objective across latent layers, one can effectively train deep-latent-variable models. This recursive structure is crucial for forcing the model to learn meaningful representations at each level of the hierarchy.
  • Optimal Transport for hierarchical learning: Optimal Transport, through the WAE framework, provides a more stable and theoretically grounded approach for training deep hierarchies compared to traditional VAEs, which often struggle with latent variable collapse.
  • Avoiding latent variable collapse: STACKEDWAE explicitly addresses the "latent variable collapse" issue, where deeper latent layers are underutilized. The proposed objective encourages the model to learn smooth distributions across the entire hierarchy.
  • Interpretable latents: The method leads to more interpretable latent representations, as evidenced by visualizations showing structured manifolds learned at each stochastic layer.

Key Results

  • Improved Sample Generation: STACKEDWAE produces qualitatively better sample generations compared to the standard WAE when training deep-hierarchical-latent models.
  • Effective Hierarchy Utilization: Experiments on MNIST, SVHN, and CelebA show that STACKEDWAE successfully utilizes all of its latent layers, capturing data structure at different levels of abstraction. Visualizations of latent spaces and reconstructions at different layers confirm this.
  • Comparison with Standard WAE: An ablation study comparing STACKEDWAE with a standard WAE on MNIST showed that the standard WAE struggled to learn smooth interpolations in the deepest latent layer, indicating poor utilization of the hierarchy, whereas STACKEDWAE performed significantly better.
  • Real-world Dataset Performance: The method demonstrated its ability to learn deep-latent representations on more complex datasets like SVHN and CelebA, even with high-dimensional latent spaces.

Limitations and Future Work

  • Latent Dimension Mismatch: The authors note that in some cases, particularly with high-dimensional latent spaces, there can be a mismatch between the intrinsic dimension of the data and the chosen latent dimension. This can lead to information loss during encoding, resulting in smoother or blurrier reconstructions as information is filtered up the hierarchy. They suggest this is a potential explanation for the observed phenomenon.
  • Tuning Hyperparameters: While the method is presented as more general, the paper still involves tuning hyperparameters (e.g., regularization weights $\lambda_{rec}, \lambda_{match}$).
  • Future Work: The paper suggests that further exploration of different cost functions and divergence measures within the STACKEDWAE framework could be beneficial. Additionally, investigating the impact of different network architectures and exploring applications beyond image generation are potential future directions.

Summary updated

Abstract

Probabilistic models with hierarchical-latent-variable structures provide state-of-the-art results amongst non-autoregressive, unsupervised density-based models. However, the most common approach to training such models based on Variational Autoencoders (VAEs) often fails to leverage deep-latent hierarchies; successful approaches require complex inference and optimisation schemes. Optimal Transport is an alternative, non-likelihood-based framework for training generative models with appealing theoretical properties, in principle allowing easier training convergence between distributions. In this work we propose a novel approach to training models with deep-latent hierarchies based on Optimal Transport, without the need for highly bespoke models and inference networks. We show that our method enables the generative model to fully leverage its deep-latent hierarchy, avoiding the well known "latent variable collapse" issue of VAEs; therefore, providing qualitatively better sample generations as well as more interpretable latent representation than the original Wasserstein Autoencoder with Maximum Mean Discrepancy divergence.
BibTeX
@misc{gaujac2020learning,
  author = {Benoit Gaujac and Ilya Feige and David Barber},
  title = {{Learning Deep-Latent Hierarchies by Stacking Wasserstein Autoencoders}},
  year = {2020},
  eprint = {2010.03467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.03467},
}
Figure