Isolating Sources of Disentanglement in Variational Autoencoders

Tian Qi Chen, Xuechen Li, Roger B. Grosse, David K. DuvenaudNeurIPS 2018 · OpenReview

Overview

This paper reveals that the success of disentangled representation learning in Variational Autoencoders (VAEs) stems specifically from penalizing Total Correlation—the redundancy between latent variables. To leverage this finding, the authors introduce $\beta$-TCVAE, a model that targets this term directly. Furthermore, they propose the Mutual Information Gap (MIG), a robust, information-theoretic metric for measuring how well a model has actually separated independent factors of variation.

The Problem with $\beta$-VAE

In the quest for "disentangled" representations—where a single latent unit corresponds to a single physical attribute like "object color" or "rotation"—the $\beta$-VAE was a major breakthrough. By increasing the penalty on the KL-divergence term in the VAE objective, researchers found they could force the model to learn independent factors.

However, the community didn't fully understand why this worked. The standard KL penalty in a $\beta$-VAE is a blunt instrument: it penalizes the total information capacity of the latents. This often leads to a trade-off where the model achieves disentanglement only by sacrificing reconstruction quality and ignoring useful data. The authors set out to perform "ELBO surgery" to find the specific component of the objective responsible for disentanglement, aiming to isolate the benefits without the drawbacks.

Methods: ELBO Surgery and $\beta$-TCVAE

The authors' primary contribution is a mathematical decomposition of the VAE's Evidence Lower Bound (ELBO). They break the standard KL term into three distinct parts:

  1. Index-Code Mutual Information: The information shared between the data and the latent variables (which we generally want to preserve).
  2. Total Correlation (TC): A measure of dependence between the latent dimensions (which we want to minimize for disentanglement).
  3. Dimension-wise KL: The divergence of each individual latent dimension from the prior.

Based on this decomposition, they propose $\beta$-TCVAE. Unlike the standard $\beta$-VAE, which penalizes all three terms equally, $\beta$-TCVAE applies the heavy $\beta$ penalty only to the Total Correlation term.

To make this practical during training, they introduce a minibatch importance sampling estimator. Calculating the "aggregated posterior" (the distribution of latents over the whole dataset) is usually computationally impossible; this estimator allows them to approximate it accurately using only small batches of data.

Finally, they propose the Mutual Information Gap (MIG). Unlike previous metrics that relied on training auxiliary classifiers, MIG is a "classifier-free" metric. It measures the difference in mutual information between a ground-truth factor and its two most informative latent dimensions, ensuring that a factor is captured by exactly one latent variable.

Key Insights and Results

The models were evaluated on several standard benchmarks, including dSprites (2D shapes), 3D Faces, 3D Chairs, and CelebA (real-world celebrity photos).

  • Disentanglement $\neq$ Information Loss: The success of $\beta$-VAE was somewhat accidental. It encouraged disentanglement by penalizing TC, but it needlessly penalized the Mutual Information between the data and the latents. $\beta$-TCVAE demonstrates that you can achieve disentanglement without losing as much information.
  • Quantitative Performance & Correlation: $\beta$-TCVAE consistently achieved higher MIG scores than $\beta$-VAE and InfoGAN across datasets like dSprites and 3D Faces. Crucially, the authors found a very strong negative correlation (e.g., $\rho = -0.95$) between Total Correlation and the MIG score, empirically proving that reducing TC is the true engine of disentanglement.
Correlation between Total Correlation and MIG, alongside quantitative disentanglement scores.
  • Qualitative Discovery: On the complex CelebA dataset, $\beta$-TCVAE successfully isolated subtle features. For example, it could extrapolate specific traits like "mustaches" or "baldness" cleanly. In contrast, standard $\beta$-VAE often entangled these features with face width or other irrelevant nuances.
Latent traversals demonstrating the isolated 'mustache' factor discovered by beta-TCVAE.
  • Reliability: $\beta$-TCVAE was shown to be more robust to different random initializations than its predecessors, consistently axis-aligning the latent space with true generative factors.

Limitations and Future Work

While $\beta$-TCVAE represents a significant step forward, some limitations remain. The minibatch importance sampling used to estimate the TC term is a lower bound and remains biased, though it is consistent as batch size increases. Additionally, while the model itself is unsupervised, calculating the MIG metric still requires a dataset with known ground-truth factors for evaluation. Finally, while the qualitative results on CelebA are impressive, moving toward disentangling highly complex, hierarchical structures in natural images remains an open challenge for the field.

Summary updated

Abstract

We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate the beta-TCVAE (Total Correlation Variational Autoencoder) algorithm, a refinement and plug-in replacement of the beta-VAE for learning disentangled representations, requiring no additional hyperparameters during training. We further propose a principled classifier-free measure of disentanglement called the mutual information gap (MIG). We perform extensive quantitative and qualitative experiments, in both restricted and non-restricted settings, and show a strong relation between total correlation and disentanglement, when the model is trained using our framework.
BibTeX
@inproceedings{chen2018isolating,
  author = {Tian Qi Chen and Xuechen Li and Roger B. Grosse and David K. Duvenaud},
  title = {{Isolating Sources of Disentanglement in Variational Autoencoders}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=HkbjqdZdbr},
}
Figure