Robust Disentanglement of a Few Factors at a Time

Benjamin Estermann, Markus Marks, Mehmet Fatih Yanik

TL;DR

The authors introduce rPU-VAE, a recursive training framework that achieves state-of-the-art unsupervised disentanglement by breaking a complex problem into smaller pieces. By identifying a few latent factors at a time, "freezing" them through data reduction, and repeating the process, the model builds a robust set of surrogate labels. These labels eventually allow an unsupervised model to perform on par with fully supervised approaches.

The Challenge of Unsupervised Disentanglement

In representation learning, disentanglement is a highly sought-after property: we want a model to learn independent latent variables that correspond to real-world generative factors (e.g., one variable for an object's x-position, another for its color, another for its rotation). Disentangled representations improve generalization, interpretability, and performance in downstream tasks.

While supervised models achieve this well, unsupervised models are notoriously finicky. Previous research (notably Locatello et al., 2019) suggested that purely unsupervised disentanglement is theoretically impossible without inductive biases. In practice, current methods suffer from extreme hyperparameter sensitivity—a model might work perfectly with one random seed and fail completely with another. Furthermore, unsupervised models often "stall," successfully capturing only the most obvious factors of variation while ignoring subtler ones.

Methodology: A Three-Stage Strategy

The authors propose a novel strategy to overcome these hurdles, culminating in the rPU-VAE architecture:

  1. Population-Based Training (PBT): Instead of training a single model, the authors train a population of models. Models that perform poorly are periodically replaced by "mutated" versions of high-performing models. This evolutionary approach makes the training process significantly more robust to hyperparameter choices and random initialization.
  2. Unsupervised Disentanglement Ranking (UDR): Because the training is unsupervised (lacking ground-truth labels), the authors use UDR as the "fitness score" for the PBT process. UDR measures how similar the learned representations are across different models in the population. The underlying assumption is that if multiple models independently discover the same latent structure, that structure is likely a true generative factor.
  3. Recursive Data Reduction (The "r" in rPU-VAE): This is the core innovation of the paper. The authors observed that models often consistently learn only the factors with the highest variance. To uncover the "hidden" factors, the algorithm employs a recursive divide-and-conquer approach:
  4. Train a model until convergence on a subset of factors.
  5. Identify the "active" latent variables.
  6. Reduce the dataset to a subset where these learned factors are held constant (using a derivative-based approach on the latent encodings).
  7. Repeat the process on this smaller dataset to force the model to find new, subtler factors.
  8. Final Step: All discovered factors are aggregated to create "surrogate labels" for the entire dataset. These labels are then used to train a final, high-performing model in a supervised manner.

Datasets

The model was evaluated on standard disentanglement benchmarks:

  • dsprites: 2D shapes (heart, oval, square) varying in scale, rotation, and x/y position.
  • shapes3d: 3D objects in a room with varying floor/wall/object colors, scales, and rotations.
  • celebA: A real-world dataset of celebrity faces used for qualitative evaluation of the model's ability to handle naturalistic data.

Key Results and Insights

  • State-of-the-Art Performance: rPU-VAE significantly outperformed existing unsupervised methods like $\beta$-TCVAE and FactorVAE on both the MIG (Mutual Information Gap) and DCI (Disentanglement, Completeness, Informativeness) metrics.
  • Unprecedented Robustness: Perhaps the most striking result is the massive reduction in variance across training runs. While previous methods showed a wide spread of performance across different random seeds, the PBT-driven approach yields consistently high results, nearly reaching the performance of models trained with full human supervision.
Comparison of disentanglement metrics showing the improved performance and reduced variance of the PBT approach compared to baselines.
  • Qualitative Success: On the shapes3d dataset, the model successfully separated wall color, floor color, and object color into distinct latent dimensions, whereas baseline models often entangled these factors.
  • Unsupervised to Supervised Bridge: The paper successfully demonstrates that an imperfect unsupervised process can be leveraged to generate "surrogate labels." Once these labels are established, the stability of supervised learning techniques can be used to train the final, robust model.

Limitations and Future Work

While highly effective, the rPU-VAE framework has a few limitations:

  • Computational Cost: Using Population-Based Training combined with multiple recursive "leaf-runs" is computationally expensive. Training a full rPU-VAE requires training many more model instances than a standard VAE.
  • Complexity of Data Reduction: The method for selecting intervals of "least variation" to reduce the dataset relies on calculating derivatives of latent encodings. This heuristic might become difficult to tune as the complexity of the data increases.
  • Scaling to Natural Images: While the authors showed promising results on celebA, the recursive logic assumes that factors can be cleanly "held constant." In highly complex, non-synthetic images, defining what it means to "remove a factor" is a much more challenging open problem.

Summary updated

Abstract

Disentanglement is at the forefront of unsupervised learning, as disentangled representations of data improve generalization, interpretability, and performance in downstream tasks. Current unsupervised approaches remain inapplicable for real-world datasets since they are highly variable in their performance and fail to reach levels of disentanglement of (semi-)supervised approaches. We introduce population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) and demonstrate the validity of this approach in a supervised setting (PBT-VAE). We then use Unsupervised Disentanglement Ranking (UDR) as an unsupervised heuristic to score models in our PBT-VAE training and show how models trained this way tend to consistently disentangle only a subset of the generative factors. Building on top of this observation we introduce the recursive rPU-VAE approach. We train the model until convergence, remove the learned factors from the dataset and reiterate. In doing so, we can label subsets of the dataset with the learned factors and consecutively use these labels to train one model that fully disentangles the whole dataset. With this approach, we show striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics.
BibTeX
@inproceedings{estermann2020robust,
  author = {Benjamin Estermann and Markus Marks and Mehmet Fatih Yanik},
  title = {{Robust Disentanglement of a Few Factors at a Time}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=9LAsnJC4p5},
}
Figure