A Heuristic for Unsupervised Model Selection for Variational Disentangled Representation Learning.

Sunny Duan, Nicholas Watters, Loic Matthey, Christopher P. Burgess, Alexander Lerchner, Irina Higgins

Abstract

Disentangled representation learning aims to isolate independent factors of variation in data—such as color, shape, and rotation—into distinct latent variables. However, the sensitivity of models like Variational Autoencoders (VAEs) to hyperparameters necessitates robust model selection, which traditionally relies on supervised metrics requiring ground truth labels. This paper introduces Unsupervised Disentanglement Ranking (UDR), a heuristic that identifies high-quality representations through pairwise comparisons between models. UDR leverages the observation that well-disentangled models converge to consistent representations, whereas entangled models fail inconsistently. Our results demonstrate that UDR correlates strongly with supervised metrics and effectively guides hyperparameter tuning across diverse datasets without requiring labels.

Introduction

Disentangled representations are highly valued in machine learning for their potential to improve AI fairness, data efficiency, and interpretability. Despite these benefits, a significant challenge remains: model performance is highly sensitive to hyperparameter settings, and selecting the "best" model typically requires ground truth attribute labels. In practical, large-scale applications, such labels are rarely available, creating a bottleneck for the deployment of disentangled learning.

The authors propose a solution based on a variation of the opening line of Anna Karenina: "Disentangled representations are all alike; every entangled representation is entangled in its own way." This heuristic suggests that models capturing the true underlying structure of the data will converge to similar latent representations (up to sign or permutation), while poorly performing models will exhibit unique, inconsistent patterns of entanglement.

Methods: Unsupervised Disentanglement Ranking (UDR)

The UDR algorithm provides a framework for ranking models within a pool without external supervision. The process involves four key steps:

  1. Model Training: A pool of models is trained using various hyperparameter settings and multiple random seeds.
  2. Pairwise Comparison: For each model, comparisons are made against other models trained with identical hyperparameters but different initial seeds.
  3. Similarity Matrix Calculation: A similarity matrix is computed between the latent dimensions of the paired models. The paper evaluates two variants: UDR<sub>L</sub>, which utilizes Lasso regression, and UDR<sub>S</sub>, which employs Spearman correlation.
  4. Score Aggregation: These similarities are aggregated into a final UDR score. A high score signifies that the model's latent dimensions have a clear, one-to-one correspondence with the latents of other models in the pool, indicating stable disentanglement.

This approach is theoretically supported by the "Polarised Regime" in VAEs, where the decoder's Jacobian tends toward orthogonal columns, creating a natural attractor for disentangled representations.

Experimental Validation and Results

The authors validated UDR using over 5,400 models trained on datasets including dSprites, 3D Shapes, 3D Cars, CelebA, and ImageNet.

Quantitative Performance

UDR scores show a strong correlation with established supervised metrics, such as the Mutual Information Gap (MIG) and the FactorVAE metric, achieving Spearman correlation coefficients of approximately 0.6. This allows UDR to effectively identify optimal hyperparameter ranges across various model architectures, including $\beta$-VAE, FactorVAE, and TC-VAE.

Quantitative comparison of UDR scores against supervised metrics across different VAE architectures and datasets (dSprites and 3D Shapes).

Qualitative Analysis

Visual inspection of latent traversals confirms the metric's efficacy. Models with high UDR scores demonstrate semantically consistent changes when individual latent dimensions are varied (e.g., changing only the rotation or color of an object). In contrast, low-scoring models exhibit entangled traversals where multiple attributes change simultaneously.

Latent traversals for a high-scoring UDR model (left) versus a low-scoring model (right) on the 3D Cars dataset. High UDR scores correspond to clear, independent factors of variation.

Discussion and Limitations

While UDR offers a powerful tool for unsupervised model selection, it has certain limitations. The requirement to train multiple seeds per hyperparameter setting increases computational costs. Additionally, the current similarity measures (Lasso and Spearman) assume continuous and monotonic factors, which may not be suitable for discrete data. Finally, UDR is a relative metric; its reliability depends on the overall quality of the model pool. Despite these challenges, UDR bridges a critical gap, enabling the practical application of disentangled representation learning in unlabeled, real-world environments.

Summary updated

Abstract

Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations to more complex domains and practical applications, it is important to enable hyperparameter tuning and model selection of existing unsupervised approaches without requiring access to ground truth attribute labels, which are not available for most datasets. This paper addresses this problem by introducing a simple yet robust and reliable method for unsupervised disentangled model selection. Our approach, Unsupervised Disentanglement Ranking (UDR), leverages the recent theoretical results that explain why variational autoencoders disentangle (Rolinek et al, 2019), to quantify the quality of disentanglement by performing pairwise comparisons between trained model representations. We show that our approach performs comparably to the existing supervised alternatives across 5,400 models from six state of the art unsupervised disentangled representation learning model classes. Furthermore, we show that the ranking produced by our approach correlates well with the final task performance on two different domains.
BibTeX
@inproceedings{duan2020heuristic,
  author = {Sunny Duan and Nicholas Watters and Loic Matthey and Christopher P. Burgess and Alexander Lerchner and Irina Higgins},
  title = {{A Heuristic for Unsupervised Model Selection for Variational Disentangled Representation Learning.}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=STf-iibmBgq},
}
Figure