Abstract
Disentanglement is at the forefront of unsupervised learning, as disentangled representations of data improve generalization, interpretability, and performance in downstream tasks. Current unsupervised approaches remain inapplicable for real-world datasets since they are highly variable in their performance and fail to reach levels of disentanglement of (semi-)supervised approaches. We introduce population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) and demonstrate the validity of this approach in a supervised setting (PBT-VAE). We then use Unsupervised Disentanglement Ranking (UDR) as an unsupervised heuristic to score models in our PBT-VAE training and show how models trained this way tend to consistently disentangle only a subset of the generative factors. Building on top of this observation we introduce the recursive rPU-VAE approach. We train the model until convergence, remove the learned factors from the dataset and reiterate. In doing so, we can label subsets of the dataset with the learned factors and consecutively use these labels to train one model that fully disentangles the whole dataset. With this approach, we show striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics.
Introduction
Disentangled representation learning aims to separate independent "factors of variation" in data—such as an object's color, shape, and position—into distinct dimensions of a latent space. While highly desirable for interpretability and downstream tasks, unsupervised disentanglement is notoriously difficult. Current methods (like $\beta$-VAE or FactorVAE) suffer from two major issues: extreme hyperparameter sensitivity and the "all-at-once" failure, where models struggle to capture all factors simultaneously, often getting stuck in local minima.
Methods
The proposed rPU-VAE framework addresses these issues through a recursive three-stage approach:
- PBT-VAE: Population Based Training (PBT) is applied to VAEs to evolve a population of models, automatically discovering optimal hyperparameter schedules (e.g., learning rate and $\beta$ annealing) during a single run.
- Unsupervised Disentanglement Ranking (UDR): To guide PBT without ground-truth labels, UDR is used as a heuristic. It assumes that well-disentangled models produce consistent representations across different initializations.
- Recursive Training (rPU-VAE): Recognizing that models often disentangle only a subset of factors (usually high-variance ones), the authors introduce a recursive loop. Dominant factors are learned, used to create "surrogate labels," and then held constant in a reduced dataset for the next iteration. Finally, a single model is trained semi-supervised using all accumulated labels.
Key Results
The rPU-VAE approach was evaluated on standard benchmarks like dsprites and shapes3d, as well as the naturalistic celebA dataset.
Quantitative Performance and Robustness
The framework achieved state-of-the-art performance, significantly closing the gap between unsupervised and fully supervised models. Notably, PBT-based training demonstrated much higher consistency and lower variance across random seeds compared to standard VAEs.
Qualitative Disentanglement
Qualitative analysis via latent traversals shows that the model successfully isolates independent factors. For example, in the celebA dataset, it isolated complex features like skin color and head rotation without human intervention.
Discussion
The "divide and conquer" strategy of rPU-VAE turns a complex unsupervised problem into a series of simpler tasks. By introducing inductive bias through the recursive procedure, the model effectively focuses on remaining variance after dominant patterns are removed. While computationally intensive due to population-based training and multiple recursions, the approach offers a significant leap in robustness and performance for unsupervised disentanglement. Future work may focus on more efficient data masking techniques to reduce the computational overhead of dataset reduction.
Summary updated