Unsupervised Model Selection for Variational Disentangled Representation Learning

Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Christopher Burgess, Alexander Lerchner, Irina Higgins

Introduction

Disentangled representation learning aims to create AI models that understand independent "factors of variation"—for example, separating an object's shape from its color or rotation. While these representations have been shown to improve fairness, data efficiency, and generalization in simple tasks, they face a major practical hurdle: model selection.

Currently, measuring how "disentangled" a model is requires access to ground-truth attribute labels (e.g., knowing exactly what the "true" rotation value is for every image). In real-world applications, these labels are rarely available. Without an unsupervised way to rank models, practitioners are left guessing which hyperparameters (like the $\beta$ in $\beta$-VAE) will work best for their specific, unlabeled dataset.

To bridge the gap between theoretical research and practical application, this paper introduces Unsupervised Disentanglement Ranking (UDR), a method to select the best disentangled models without needing ground-truth labels.

The UDR Method: Consistency as Quality

The core intuition behind UDR is inspired by the opening line of Tolstoy's Anna Karenina: "Happy families are all alike; every unhappy family is unhappy in its own way."

In the context of representation learning:

  1. Disentangled models are "happy": If multiple models successfully disentangle a dataset, they should all converge to the same underlying representation (up to a simple reordering or sign flip of the latent dimensions).
  2. Entangled models are "unhappy": Models that fail to disentangle will fail in unique, inconsistent ways based on their random initialization.
Latent traversals for 3D Cars showing that high UDR scores correspond to consistent disentanglement across different random seeds, while low UDR scores indicate entanglement.

The UDR Algorithm: 1. Train: Train several models with different hyperparameters and random seeds. 2. Compare: Perform pairwise comparisons between the trained models. 3. Compute Similarity: Calculate a similarity matrix between the latent responses of two models using either Lasso regression or Spearman correlation. 4. Score: Aggregate these similarities into a single UDR score. A high score means the model's representation is consistently found across other training runs, suggesting it has captured the "true" latent structure of the data.

This approach avoids "canonical" bias. Supervised metrics often penalize a model if it doesn't align with a specific human-chosen coordinate system (e.g., representing color as RGB vs. HSV). UDR is more flexible; it accepts any representation as long as it is consistently discovered.

Key Results and Validation

The authors validated UDR across a wide range of benchmarks, including synthetic datasets with ground truth (dSprites, 3D Shapes) and complex real-world datasets (3D Cars, CelebA, ImageNet).

Strong Correlation with Supervised Metrics UDR scores correlate remarkably well with established supervised metrics (like the FactorVAE metric or MIG). Across thousands of models from six different state-of-the-art model classes (including $\beta$-VAE, FactorVAE, and DIP-VAE), UDR achieved Spearman rank correlations around 0.6 to 0.7.

Comparison of UDR (highlighted) against supervised metrics across various VAE models on dSprites and 3D Shapes datasets, demonstrating strong correlation.

Predicting Downstream Task Performance Crucially, UDR is not just a mathematical curiosity; it effectively predicts how well a model will perform on downstream tasks. The ranking produced by UDR showed an 0.8 correlation with fairness scores and a 0.56 correlation with data efficiency in reinforcement learning tasks. Furthermore, the method remains robust and reliable even with as few as five pairwise comparisons.

Limitations and Future Work

While UDR provides a powerful tool for unsupervised model selection, it does have limitations:

  • Computational Cost: UDR requires training multiple "seeds" (different random initializations) for each hyperparameter setting to perform the pairwise comparisons. This can be computationally expensive for very large models.
  • The "Herd Effect": Because UDR relies on relative comparisons, a model's score depends on the quality of the other models in the pool. If all models in a sweep are poor, the ranking might become less reliable.
  • Assumption of Monotonicity: The current implementation assumes that the relationship between latent dimensions is continuous and monotonic. Future work could adapt UDR for more complex, non-geometric factors of variation.

Ultimately, UDR represents a significant step forward in making disentangled representation learning viable for real-world, unlabeled datasets.

Summary updated

Abstract

Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations to more complex domains and practical applications, it is important to enable hyperparameter tuning and model selection of existing unsupervised approaches without requiring access to ground truth attribute labels, which are not available for most datasets. This paper addresses this problem by introducing a simple yet robust and reliable method for unsupervised disentangled model selection. We show that our approach performs comparably to the existing supervised alternatives across 5400 models from six state of the art unsupervised disentangled representation learning model classes. Furthermore, we show that the ranking produced by our approach correlates well with the final task performance on two different domains.
BibTeX
@inproceedings{duan2020unsupervised,
  author = {Sunny Duan and Loic Matthey and Andre Saraiva and Nick Watters and Christopher Burgess and Alexander Lerchner and Irina Higgins},
  title = {{Unsupervised Model Selection for Variational Disentangled Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=STf-iibmBgq},
}
Figure