Robust Disentanglement of a Few Factors at a Time

Benjamin Estermann, Markus Marks, Mehmet Fatih Yanik

TL;DR

Unsupervised disentanglement models are notoriously unstable and highly sensitive to hyperparameter choices. This paper introduces rPU-VAE, a recursive approach that leverages Population-Based Training (PBT) and Unsupervised Disentanglement Ranking (UDR) to consistently isolate a few dominant factors of variation at a time. By iteratively removing the variance of discovered factors from the dataset, the algorithm progressively uncovers all underlying generative factors, matching the performance of supervised methods without requiring human labels.

What Problem Does This Paper Address?

Disentangled representation learning aims to identify independent, low-dimensional generative factors (like position, scale, or color) from high-dimensional data (like images). While supervised approaches excel at this, unsupervised methods are plagued by extreme hyperparameter sensitivity and high performance variance across different random seeds. In fact, prominent theoretical work suggests that fully unsupervised disentanglement is impossible without some form of inductive bias.

Furthermore, the authors observe a specific failure mode in existing unsupervised models: they tend to only capture a subset of the generative factors—typically the ones causing the highest variance in the image space—and completely ignore more subtle factors during training.

Methods

To overcome these challenges, the authors propose a multi-step framework:

  1. Population-Based Training (PBT): Instead of training a single model with fixed hyperparameters, PBT trains a population of Variational Autoencoders (VAEs) simultaneously. High-performing models are cloned ("exploit"), and their hyperparameters (such as learning rate, batch size, and the regularization coefficient $\beta$) are randomly perturbed ("explore") during training.
  2. Unsupervised Disentanglement Ranking (UDR): In the unsupervised setting, the authors use UDR as the fitness metric to guide PBT. UDR measures the representational similarity between different models trained with the same hyperparameters, acting as a robust unsupervised heuristic.
  3. Recursive Data Reduction (rPU-VAE): Since the PBT-U-VAE consistently plateaus after learning only the most dominant factors, the authors turn this limitation into a feature. Once the model converges on a subset of factors, the entire dataset is mapped onto these active latents. The authors compute the derivative of these latent encodings to find intervals where the factors change the least, effectively filtering the dataset to remove their variance.

This data-reduction process is repeated recursively (termed "leaf-runs") to discover less dominant factors. The discovered factors are used to generate "surrogate labels" for the dataset, which are finally fed into a semi-supervised PBT-VAE to train a single model that fully disentangles the entire dataset.

Figure 1: Developing Population Based Training for (semi-)supervised disentanglement. a) MIG is the preferred eval function for PBT-S-VAE, as it outperforms DCI Disentanglement when measured with both metrics respectively. b) The semi-supervised 1000-PBT-S-VAE (MIG) outperforms both Locatello et al. [15] and the results of the model selection when all of them are trained on 1000 dsprites labels. The fully supervised PBT-S-VAE further increases performance in comparison to the semi-supervised approaches.

Datasets

The method was evaluated on three widely used benchmarks:

  • dsprites: A synthetic dataset of 2D shapes varying by x/y position, scale, rotation, and shape.
  • shapes3d: A 3D rendered dataset varying by floor/wall/object color, object size, type, and azimuth.
  • celebA: A complex, real-world dataset of celebrity faces containing various facial attributes.

Key Insights and Ideas

  • Exploiting a Limitation: The core "aha moment" of the paper is realizing that unsupervised models naturally prioritize high-variance factors and stop exploring new ones. Rather than trying to force a single model to learn everything at once, rPU-VAE embraces this behavior by peeling away layers of variance sequentially.
  • Dynamic Hyperparameter Schedules: PBT automatically discovers optimal, non-linear training schedules. For instance, it naturally learns to anneal the $\beta$ parameter (which controls the disentanglement bottleneck) over time, a strategy that is difficult to tune manually.
  • Surrogate Supervision: The framework bridges the gap between unsupervised and supervised learning by using intermediate unsupervised models to auto-label the dataset, providing a clean supervision signal for the final model.

Key Results

The recursive rPU-VAE approach demonstrates substantial improvements in both performance and stability:

  • State-of-the-Art Performance: On both dsprites and shapes3d, rPU-VAE significantly outperforms standard unsupervised baselines like $\beta$-VAE, FactorVAE, and $\beta$-TCVAE when measured via Mutual Information Gap (MIG) and DCI Disentanglement metrics.
  • Drastic Variance Reduction: Unlike traditional methods where performance is a roll of the dice based on the random seed, rPU-VAE exhibits remarkably low variance, approaching the upper-bound performance of fully supervised models.
Performance comparison of rPU-VAE against state-of-the-art unsupervised models
  • Real-World Scaling: When applied to the complex celebA dataset, the model successfully disentangled distinct, interpretable factors such as skin color, background color, head rotation, and hair orientation without human annotations.

Limitations and Future Work

The primary drawback of rPU-VAE is its computational complexity. Training a large population of models (the authors used a population size of 56) across multiple recursive leaf-runs requires substantial computational resources and energy.

The authors acknowledge that this high energy consumption translates to a significant carbon footprint. They suggest that future work should explore more computationally efficient paradigms, such as spiking neural networks, to make this recursive population-based approach more sustainable.

Summary updated

Abstract

Disentanglement is at the forefront of unsupervised learning, as disentangled representations of data improve generalization, interpretability, and performance in downstream tasks. Current unsupervised approaches remain inapplicable for real-world datasets since they are highly variable in their performance and fail to reach levels of disentanglement of (semi-)supervised approaches. We introduce population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) and demonstrate the validity of this approach in a supervised setting (PBT-VAE). We then use Unsupervised Disentanglement Ranking (UDR) as an unsupervised heuristic to score models in our PBT-VAE training and show how models trained this way tend to consistently disentangle only a subset of the generative factors. Building on top of this observation we introduce the recursive rPU-VAE approach. We train the model until convergence, remove the learned factors from the dataset and reiterate. In doing so, we can label subsets of the dataset with the learned factors and consecutively use these labels to train one model that fully disentangles the whole dataset. With this approach, we show striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics.
BibTeX
@inproceedings{estermann2020robust,
  author = {Benjamin Estermann and Markus Marks and Mehmet Fatih Yanik},
  title = {{Robust Disentanglement of a Few Factors at a Time}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=9LAsnJC4p5},
}
Figure