Abstract
Disentangled representation learning aims to isolate independent factors of variation in data—such as color, shape, and rotation—into distinct latent variables. However, the sensitivity of models like Variational Autoencoders (VAEs) to hyperparameters necessitates robust model selection, which traditionally relies on supervised metrics requiring ground truth labels. This paper introduces Unsupervised Disentanglement Ranking (UDR), a heuristic that identifies high-quality representations through pairwise comparisons between models. UDR leverages the observation that well-disentangled models converge to consistent representations, whereas entangled models fail inconsistently. Our results demonstrate that UDR correlates strongly with supervised metrics and effectively guides hyperparameter tuning across diverse datasets without requiring labels.
Introduction
Disentangled representations are highly valued in machine learning for their potential to improve AI fairness, data efficiency, and interpretability. Despite these benefits, a significant challenge remains: model performance is highly sensitive to hyperparameter settings, and selecting the "best" model typically requires ground truth attribute labels. In practical, large-scale applications, such labels are rarely available, creating a bottleneck for the deployment of disentangled learning.
The authors propose a solution based on a variation of the opening line of Anna Karenina: "Disentangled representations are all alike; every entangled representation is entangled in its own way." This heuristic suggests that models capturing the true underlying structure of the data will converge to similar latent representations (up to sign or permutation), while poorly performing models will exhibit unique, inconsistent patterns of entanglement.
Methods: Unsupervised Disentanglement Ranking (UDR)
The UDR algorithm provides a framework for ranking models within a pool without external supervision. The process involves four key steps:
- Model Training: A pool of models is trained using various hyperparameter settings and multiple random seeds.
- Pairwise Comparison: For each model, comparisons are made against other models trained with identical hyperparameters but different initial seeds.
- Similarity Matrix Calculation: A similarity matrix is computed between the latent dimensions of the paired models. The paper evaluates two variants: UDR<sub>L</sub>, which utilizes Lasso regression, and UDR<sub>S</sub>, which employs Spearman correlation.
- Score Aggregation: These similarities are aggregated into a final UDR score. A high score signifies that the model's latent dimensions have a clear, one-to-one correspondence with the latents of other models in the pool, indicating stable disentanglement.
This approach is theoretically supported by the "Polarised Regime" in VAEs, where the decoder's Jacobian tends toward orthogonal columns, creating a natural attractor for disentangled representations.
Experimental Validation and Results
The authors validated UDR using over 5,400 models trained on datasets including dSprites, 3D Shapes, 3D Cars, CelebA, and ImageNet.
Quantitative Performance
UDR scores show a strong correlation with established supervised metrics, such as the Mutual Information Gap (MIG) and the FactorVAE metric, achieving Spearman correlation coefficients of approximately 0.6. This allows UDR to effectively identify optimal hyperparameter ranges across various model architectures, including $\beta$-VAE, FactorVAE, and TC-VAE.
Qualitative Analysis
Visual inspection of latent traversals confirms the metric's efficacy. Models with high UDR scores demonstrate semantically consistent changes when individual latent dimensions are varied (e.g., changing only the rotation or color of an object). In contrast, low-scoring models exhibit entangled traversals where multiple attributes change simultaneously.
Discussion and Limitations
While UDR offers a powerful tool for unsupervised model selection, it has certain limitations. The requirement to train multiple seeds per hyperparameter setting increases computational costs. Additionally, the current similarity measures (Lasso and Spearman) assume continuous and monotonic factors, which may not be suitable for discrete data. Finally, UDR is a relative metric; its reliability depends on the overall quality of the model pool. Despite these challenges, UDR bridges a critical gap, enabling the practical application of disentangled representation learning in unlabeled, real-world environments.
Summary updated