Gaussian mixture models with Wasserstein distance

Benoit Gaujac, Ilya Feige, David BarberarXiv 2018 · OpenReview

TL;DR

This paper tackles the challenge of training generative models that combine discrete and continuous latent variables, which are crucial for modeling real-world data but notoriously difficult to train. By leveraging the Optimal Transport framework, specifically Wasserstein Autoencoders, the authors demonstrate a significantly more stable and effective training process, allowing discrete latent variables to be fully utilized and leading to better sample generation and control.

What Problem Does This Paper Address?

Many real-world datasets exhibit a hierarchical structure, where data can be naturally categorized into distinct groups (discrete latent variables) and then described by continuous variations within those groups (continuous latent variables). Models that capture this structure, like Gaussian Mixture Latent Variable Models (GM-LVMs), are highly desirable for unsupervised learning. However, training these models, particularly when using traditional Variational Autoencoder (VAE) objectives, proves problematic. The VAE objective, which relies on per-data-point optimization, causes the discrete latent variable's distribution to collapse early in training, effectively ignoring it. This leads to poor generation quality and a lack of control over the generated samples. The paper aims to overcome this training instability and unlock the full potential of these hybrid discrete-continuous generative models.

Methods

The core contribution of this paper is the application of the Optimal Transport (OT) framework, specifically the Wasserstein distance, to train Gaussian Mixture Latent Variable Models (GM-LVMs). This approach is termed Gaussian-Mixture Wasserstein Autoencoders (GM-WAE).

Key aspects of their method include:

  • Wasserstein Distance Objective: Instead of the VAE's evidence lower bound (ELBO), they minimize the 2-Wasserstein distance between the data distribution and the generative model distribution. This distance is known to induce a weaker topology on the space of distributions, facilitating easier convergence.
  • Relaxed Wasserstein Distance: To make the optimization tractable, they use a relaxed version of the Wasserstein distance that incorporates a Maximum Mean Discrepancy (MMD) term.
  • MMD with IMQ Kernels: For the MMD, they employ a mixture of inverse multiquadratic (IMQ) kernels with varying bandwidths. IMQ kernels are chosen for their fatter tails, which are beneficial during early training.
  • Deterministic Generative Model: Unlike VAEs, the Wasserstein distance framework allows for deterministic generative models, where the data x is a direct function g(z) of the continuous latent z. This is implemented using a deep neural network.
  • Parametrization of Variational Distribution: The variational distribution q(z, k|x) is parameterized to mirror the prior, with a Gaussian for the continuous latent qc(z|k,x) and a categorical distribution for the discrete latent qD(k|x). The reparameterization trick is used for the continuous latent.
  • Exact Expectation for Discrete Latent: To avoid issues with backpropagating through discrete variables, they compute the expectation over the discrete latent distribution exactly, which is feasible for a small number of discrete classes.

Datasets

The paper uses the MNIST dataset for training and evaluation. No new benchmarks are introduced.

Key Insights and Ideas

  • Wasserstein Distance as a Training Stabilizer: The most significant insight is that the Wasserstein distance objective fundamentally changes the training dynamics, preventing the collapse of the discrete latent variable that plagues VAEs. This is attributed to the aggregated nature of the Wasserstein distance, which doesn't demand per-data-point similarity.
  • Discrete Latent's Full Leverage: The GM-WAE successfully learns to utilize its discrete latent variable, allowing it to capture distinct classes of data (e.g., digits) effectively. This contrasts sharply with GM-VAEs, where the discrete latent is often ignored.
  • Improved Generation Quality and Control: By effectively leveraging the discrete latent, the GM-WAE generates more coherent and representative samples. Furthermore, the structured latent space provides a greater degree of control over generation, allowing for interpolations that correspond to meaningful semantic changes.
  • Simpler Architectures Suffice: The paper suggests that the improved training stability and latent structure allow for relatively simple neural network architectures to achieve competitive results, as the discrete latent carries a significant portion of the descriptive burden.

Key Results

  • Successful Training on MNIST: The GM-WAE is successfully trained on MNIST in an unsupervised setting, demonstrating the effectiveness of the Wasserstein distance for GM-LVMs.
  • Competitive Sample Quality: The generated samples are qualitatively competitive with other state-of-the-art approaches.
  • Meaningful Latent Space Interpolations: Linear interpolations in the continuous latent space between data points and between prior modes result in smooth and semantically meaningful transitions in the generated digits.
  • Digit-Class Assignment Accuracy: The discrete encoder achieves a digit-class assignment accuracy of 76% on the held-out test set, which is respectable given the unsupervised nature and relatively simple model.
  • VAE Failure Case: A direct comparison shows that initializing a VAE with the trained WAE's parameters and training with the VAE objective leads to significant deterioration in reconstruction quality, highlighting the inherent training difficulties of VAEs for this model class.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the exact infimum in the Wasserstein distance objective is approximated by parameterizing the variational distribution.
  • While the discrete expectation is computed exactly for MNIST, this is not scalable to a large number of discrete classes, and the paper notes that this can be relaxed using techniques like Gumbel-Softmax.
  • The accuracy of the digit-class assignment (76%) is lower than some adversarial autoencoder approaches (90%), though the authors attribute this to a simpler model and training procedure.
  • Future Work:
    • The authors express hope that their work will motivate further study into the application of Optimal Transport in generative modeling, particularly for complex latent structures.
  • The paper implicitly suggests exploring more sophisticated neural network architectures or training strategies to further improve performance and scalability.
  • Investigating the application of this approach to datasets beyond MNIST is a natural next step.

Summary updated

Abstract

Generative models with both discrete and continuous latent variables are highly motivated by the structure of many real-world data sets. They present, however, subtleties in training often manifesting in the discrete latent being under leveraged. In this paper, we show that such models are more amenable to training when using the Optimal Transport framework of Wasserstein Autoencoders. We find our discrete latent variable to be fully leveraged by the model when trained, without any modifications to the objective function or significant fine tuning. Our model generates comparable samples to other approaches while using relatively simple neural networks, since the discrete latent variable carries much of the descriptive burden. Furthermore, the discrete latent provides significant control over generation.
BibTeX
@misc{gaujac2018gaussian,
  author = {Benoit Gaujac and Ilya Feige and David Barber},
  title = {{Gaussian mixture models with Wasserstein distance}},
  year = {2018},
  eprint = {1806.04465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.04465},
}
Figure