InfoCatVAE: Representation Learning with Categorical Variational Autoencoders

Edouard Pineau, Marc LelargearXiv 2018 · OpenReview

TL;DR

This paper introduces InfoCatVAE, a novel variational autoencoder that learns disentangled representations by incorporating categorical latent variables and maximizing mutual information. This approach leads to more robust and interpretable representations, particularly in high-capacity regimes, and improves generative capabilities.

What Problem Does This Paper Address?

Unsupervised representation learning aims to discover meaningful underlying factors of variation in data without explicit labels. Variational Autoencoders (VAEs) are a popular framework for this, but standard VAEs with simple Gaussian priors often struggle to learn truly disentangled representations. This means that each latent dimension doesn't necessarily correspond to a single, independent factor of variation in the data. Existing methods to improve disentanglement often rely on complex architectures or heuristics. This paper addresses the challenge of achieving robust and interpretable disentangled representations in a principled way, especially when dealing with complex data distributions.

Methods

The core contribution is InfoCatVAE, an extension of the Variational Autoencoder. Here's a breakdown of the key modifications:

  • Categorical Latent Variable (CatVAE): Instead of a purely continuous latent space, CatVAE introduces a discrete categorical latent variable c (e.g., representing different classes or types of features). The prior distribution is now a mixture of Gaussians, p(z|c), conditioned on this categorical variable. This allows for learning distinct groups of factors.
  • Multimodal Prior and Inference Network: Both the prior distribution p(z|c) and the inference network q(z|x, c) are designed to be multimodal, reflecting the categorical structure.
  • Modified Evidence Lower Bound (ELBO): The ELBO is re-derived to accommodate the categorical variable. This modified ELBO naturally leads to a soft clustering objective, where data points are assigned probabilities to different categories.
  • Information Maximization (InfoCatVAE): Inspired by InfoGAN, an additional term is added to the objective function to maximize the mutual information between the categorical code c and the generated data. This encourages the model to learn more salient and informative categorical representations.
  • Fixed Multimodal Prior: A key aspect is the use of a fixed multimodal prior, where the means of the Gaussian components are orthogonal. This is argued to provide robustness, especially in high-capacity regimes, preventing the instability often seen with free multimodal priors.

Datasets

The paper uses the following datasets:

  • MNIST: A standard dataset of handwritten digits.
  • FashionMNIST: A dataset of clothing items.
  • Multivariate Sequential MNIST: A novel benchmark created by treating MNIST images as 28-dimensional time series. This is used to test the model's ability to handle more complex sequential data and hierarchical representations.

Key Insights and Ideas

  • Categorical Latent Variables for Disentanglement: Introducing a discrete categorical variable alongside a continuous latent space is a powerful way to enforce disentanglement, naturally leading to a soft clustering of data.
  • Robustness through Fixed Priors: Using a fixed, structured multimodal prior (orthogonal means) provides robustness in high-capacity regimes, preventing the learning instability that can arise with free multimodal priors. This is a key differentiator from some prior work.
  • Information Maximization Enhances Categorical Learning: Adapting the InfoGAN approach to maximize mutual information between the categorical code and generated data significantly improves the learning of salient categorical attributes.
  • Soft Clustering as a Natural Outcome: The modified ELBO for CatVAE inherently encourages a soft clustering behavior, making the learned representations more interpretable.

Key Results

  • Readable Representations: InfoCatVAE demonstrates the ability to learn readable representations, as shown through discrete interpolations between learned centroids in the latent space (Figure 3).
  • Improved Generation: The model shows improved generative capabilities, producing diverse and coherent samples.
  • Robustness in High-Capacity Regimes: Experiments on multivariate sequential MNIST show that InfoCatVAE, with its fixed multimodal prior and information maximization, is more robust and interpretable than models with free priors.
  • Quantitative Improvements: InfoCatVAE achieves higher log-likelihood scores on MNIST compared to a standard Adversarial Autoencoder, particularly when the number of categories is increased. The paper also reports improved cross-entropy scores for mutual information estimation between generated samples and categories.

Limitations and Future Work

  • Prior Choice Sensitivity: While the fixed multimodal prior offers robustness, the authors acknowledge that fixing the prior can still pose information transfer and representation problems. The choice of prior parameters is crucial.
  • Free Prior Potential: For specific tasks, a well-controlled free prior with a fine-tuned architecture might offer superior performance.
  • Future Work:
    • Bayesian Hyperparameterization: Prefixing prior parameters with Bayesian hyperparameterization is suggested to improve latent space structure while maintaining stability.
  • Exploring Different Priors: Further exploration of different prior structures and their impact on disentanglement and generation is a natural next step.

Summary updated

Abstract

This paper describes InfoCatVAE, an extension of the variational autoencoder that enables unsupervised disentangled representation learning. InfoCatVAE uses multimodal distributions for the prior and the inference network and then maximizes the evidence lower bound objective (ELBO). We connect the new ELBO derived for our model with a natural soft clustering objective which explains the robustness of our approach. We then adapt the InfoGANs method to our setting in order to maximize the mutual information between the categorical code and the generated inputs and obtain an improved model.
BibTeX
@misc{pineau2018infocatvae,
  author = {Edouard Pineau and Marc Lelarge},
  title = {{InfoCatVAE: Representation Learning with Categorical Variational Autoencoders}},
  year = {2018},
  eprint = {1806.08240},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.08240},
}

Loading related content...

Figure