Understanding disentangling in β-VAE

Christopher P. Burgess, Irina Higgins, Arka Pal, Loïc Matthey, Nick Watters, Guillaume Desjardins, Alexander LerchnerarXiv 2018 · OpenReview

Abstract

We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in $\beta$-VAE, as training progresses. From these insights, we propose a modification to the training regime of $\beta$-VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in $\beta$-VAE, without the previous trade-off in reconstruction accuracy.

Introduction

In unsupervised machine learning, learning "disentangled" representations—where single latent variables correspond to independent ground-truth factors of variation such as position or scale—is a key goal for interpretability. The $\beta$-VAE framework achieves this by weighting the Kullback-Leibler (KL) divergence term in the loss function with a hyperparameter $\beta > 1$. However, this approach has historically suffered from a lack of theoretical clarity regarding why it works and a significant "reconstruction trade-off," where high $\beta$ values lead to blurry, low-quality images.

This paper addresses these gaps by viewing the $\beta$-VAE objective through the lens of information bottleneck theory and proposing a controlled capacity increase to achieve both disentanglement and high-fidelity reconstruction.

Theoretical Framework: The Information Bottleneck

The authors analyze $\beta$-VAE as an Information Bottleneck, where the KL divergence term acts as a constraint on the information flow through the latent channel. This constraint creates "locality pressure."

Locality pressure: overlapping posterior distributions $q(z|x)$ force visually similar inputs to be mapped to nearby latent coordinates to minimize reconstruction error.

When the latent capacity is restricted, the model is forced to overlap the posterior distributions of different data points. To minimize reconstruction error despite this overlap, the model must map visually similar images to nearby points in the latent space. Because the VAE uses a diagonal covariance matrix for its posterior, it must allocate this limited capacity independently across dimensions. To optimally allocate capacity to different generative factors (e.g., more for position, less for scale), the model naturally aligns these distinct factors with orthogonal latent axes.

Proposed Method: Controlled Capacity Increase

To resolve the reconstruction trade-off, the authors propose a novel training objective. Instead of a fixed $\beta$ penalty, they introduce a target capacity parameter $C$ that is linearly increased during training. The loss function minimizes the difference between the KL divergence and this target:

$$\mathcal{L} = \mathbb{E}[\log p(x|z)] - \gamma |D_{KL}(q(z|x) || p(z)) - C|$$

This "key algorithmic tweak" gradually relaxes the information bottleneck, allowing the model to learn factors of variation sequentially while maintaining the disentanglement of previously learned factors.

Experimental Results

The method was evaluated on benchmarks including dSprites, 3D Chairs, and CelebA.

Sequential learning of factors on dSprites. As capacity $C$ (measured in nats) increases, the model learns position, then scale, shape, and finally orientation.

Sequential Learning of Factors

Empirical results demonstrate that factors are learned one by one based on their contribution to pixel-wise reconstruction. On the dSprites dataset, X/Y position emerges first (at ~3 nats), followed by scale (~7 nats), shape (~15 nats), and finally rotation (~24 nats). This confirms that the model prioritizes factors that yield the largest improvements in log-likelihood under a tight bottleneck.

Overcoming the Trade-off

Unlike the original $\beta$-VAE, which produces blurry images when disentangling, the controlled capacity increase method successfully disentangles all generative factors while producing high-quality reconstructions. The resulting images are visually difficult to distinguish from the original data samples, effectively breaking the traditional trade-off between disentanglement and reconstruction quality.

Discussion and Limitations

While the bottleneck approach is powerful, the order of factor learning is strictly dictated by their contribution to pixel-wise reconstruction log-likelihood. This means visually subtle but semantically important features might be ignored until late in training. Additionally, the method requires tuning the capacity schedule for different datasets. Future work aims to scale these insights to more complex, real-world visual datasets.

Summary updated

Abstract

We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in $\beta$-VAE, as training progresses. From these insights, we propose a modification to the training regime of $\beta$-VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in $\beta$-VAE, without the previous trade-off in reconstruction accuracy.
BibTeX
@misc{burgess2018understanding,
  author = {Christopher P. Burgess and Irina Higgins and Arka Pal and Loïc Matthey and Nick Watters and Guillaume Desjardins and Alexander Lerchner},
  title = {{Understanding disentangling in β-VAE}},
  year = {2018},
  eprint = {1804.03599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.03599},
}
Figure