Robust Disentanglement of a Few Factors at a Time using rPU-VAE

NeurIPS 2020

TL;DR

This paper introduces a novel recursive approach (rPU-VAE) to achieve robust unsupervised disentanglement of data representations. By iteratively identifying and isolating factors of variation, the method effectively labels subsets of data, enabling a more comprehensive disentanglement than previous state-of-the-art methods, even outperforming some supervised approaches.

What Problem Does This Paper Address?

Unsupervised disentanglement aims to learn representations where distinct underlying factors of variation in data (e.g., shape, color, position) are captured by separate dimensions in the learned representation. This is crucial for improving generalization, interpretability, and performance in downstream tasks. However, current unsupervised methods struggle with real-world datasets, exhibiting inconsistent performance and failing to achieve high levels of disentanglement. A major challenge is the extreme sensitivity to hyperparameters, leading to a lack of robustness. Previous work has shown that fully unsupervised disentanglement might be impossible without some form of inductive bias, leading to explorations of weak or partial supervision.

Methods

The paper builds upon Population Based Training (PBT) [20] to address hyperparameter sensitivity and achieve more consistent training of Variational Autoencoders (VAEs).

  • PBT-VAE: Initially, they adapt PBT for VAE training, demonstrating its effectiveness in supervised and semi-supervised settings.
  • Unsupervised Disentanglement Ranking (UDR): For unsupervised learning, they use UDR [8] as an evaluation metric within PBT. This leads to PBT-U-VAE (UDR), which consistently disentangles a subset of factors with the largest variance.
  • Recursive PBT-U-VAE (rPU-VAE): The core innovation is the recursive approach. After PBT-U-VAE (UDR) converges on a subset of factors, these learned factors are "removed" from the dataset by creating a reduced dataset where these variations are minimized. This process is repeated, forcing the model to learn new factors of variation in subsequent "metaEpochs."
  • Labeling and Final Model Training: The identified factors from each recursive step are used to generate "surrogate labels" for subsets of the data. Finally, a PBT-S-VAE (MIG) model is trained using these surrogate labels, effectively combining all learned factors into a single, fully disentangled representation.

Datasets

The paper uses the following datasets for training and evaluation:

  • dsprites [6]: A benchmark dataset for disentanglement research, featuring simple 2D shapes with variations in position, scale, rotation, and color.
  • shapes3d [27]: Another dataset commonly used for disentanglement evaluation, similar to dsprites but with more complex shapes and variations.
  • celebA [29]: A naturalistic dataset of celebrity faces, used to demonstrate the method's applicability to real-world, complex data.

Key Insights and Ideas

  • PBT for Robustness: Population Based Training is a powerful tool for overcoming the hyperparameter sensitivity that plagues unsupervised disentanglement, leading to more stable and consistent results.
  • UDR as an Unsupervised Metric: UDR provides a viable unsupervised heuristic for guiding PBT, enabling the identification of models that consistently disentangle a subset of factors.
  • Iterative Factor Isolation: The key insight is that by iteratively identifying and "removing" learned factors, the model can be forced to discover new, previously unlearned factors of variation. This recursive process is crucial for achieving comprehensive disentanglement.
  • Surrogate Labels Bridge Unsupervised and Supervised: The generated surrogate labels from the unsupervised recursive steps act as a bridge, enabling the training of a final supervised model that leverages the discovered disentangled factors.
  • Subset Disentanglement is a Stepping Stone: Models that consistently disentangle only a subset of factors (as observed with PBT-U-VAE (UDR)) are not necessarily worse; they represent a valuable intermediate step in a more comprehensive disentanglement strategy.

Key Results

  • State-of-the-Art Performance: rPU-VAE significantly outperforms previous unsupervised disentanglement approaches on both dsprites and shapes3d datasets, achieving higher mean MIG/DCI Disentanglement scores.
  • Drastically Reduced Variance: The proposed method exhibits significantly lower variance in disentanglement scores compared to other approaches, demonstrating its robustness.
  • Near-Supervised Performance: The unsupervised rPU-VAE achieves scores very close to their supervised PBT-S-VAE (MIG) model, which serves as an upper bound for performance.
  • Impact of Leaf-Runs: The number of leaf-runs (recursive steps) positively impacts performance, with the most significant improvement observed from 0 to 1 leaf-run. Increasing leaf-runs also further reduces variance.
  • Qualitative Results: The paper shows qualitative results on dsprites, shapes3d, and celebA, demonstrating the learned factors and their traversals, including factors like position, scale, rotation, shape, and color.

Limitations and Future Work

  • Computational Cost: The approach relies on training large numbers of models, which requires significant computational resources and has an associated environmental cost (energy consumption, CO2 footprint).
  • Potential for Over-Trust: While enhanced interpretability is a benefit, the paper notes that an undue sense of trust in disentangled representations could lead to negative consequences if failures occur. Further research on the reliability of AI approaches is needed.
  • Future Work: The authors suggest exploring more efficient computational methods, such as spiking neural networks, to reduce the environmental impact. They also highlight the need for further research on the reliability of disentangled representations.

Summary updated

BibTeX
@inproceedings{2020robust,
  title = {{Robust Disentanglement of a Few Factors at a Time using rPU-VAE}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure