A Framework for the Quantitative Evaluation of Disentangled Representations

Cian Eastwood, Christopher K. I. WilliamsICLR 2018 · OpenReview

TL;DR

This paper introduces a formal quantitative framework to measure how well machine learning models "disentangle" the underlying factors of data (such as an object's rotation versus its color). By moving beyond subjective visual inspection, the authors define three distinct metrics—disentanglement, completeness, and informativeness—allowing for a rigorous, head-to-head comparison of popular generative models like VAEs and InfoGAN.

The Problem: Moving Beyond Visual Inspection

In representation learning, the primary goal is to take raw, high-dimensional data (like pixels in an image) and compress it into a compact "code" that captures the essential explanatory factors. Ideally, this code should be disentangled: changing one variable in the code should only alter one physical aspect of the data (e.g., adjusting the lighting without changing the object's shape).

Historically, while many researchers claimed their models achieved disentanglement, they typically "proved" it by showcasing a few cherry-picked images where they tweaked a latent variable and observed the result. This process of visual inspection is highly subjective and lacks a clear, standardized definition of what makes one representation quantitatively "better" than another. This paper addresses this gap by proposing standardized metrics for evaluating these models when the ground-truth factors are known.

A Quantitative Evaluation Framework

The authors propose a framework that treats evaluation as a supervised learning problem. Given a synthetic dataset where the ground-truth generative factors ($z$) are known, a regressor can be trained to predict those factors from the model's learned latent code ($c$). By analyzing the "importance matrix" ($R$) of this regressor, the authors derive three core metrics:

  1. Disentanglement: Does each variable in the learned code ($c_i$) capture at most one generative factor ($z_j$)? This is calculated using the entropy of the rows of the importance matrix.
  2. Completeness: Is each generative factor ($z_j$) captured by at most one code variable ($c_i$)? This is calculated using the entropy of the columns of the importance matrix.
  3. Informativeness: How much information about the underlying factors is actually present in the code? This is measured by the prediction error of the regressor.
Mapping between ground-truth factors (z) and learned codes (c), illustrating disentanglement and completeness.

To capture different types of representations, the authors utilize two types of regressors: Lasso (linear and sparse) to determine if the information is "explicitly" and simply represented, and Random Forests (non-linear) to see if the information is present at all, even if it is highly "tangled."

The Teapot Dataset

To test their framework, the study uses a synthetic Teapot Dataset consisting of 200,000 64x64 color images.

Examples from the synthetic Teapot dataset showing variations in color, azimuth, and elevation.

The images vary across five independent generative factors:

  • Azimuth: Horizontal rotation ($0$ to $2\pi$).
  • Elevation: Vertical rotation ($0$ to $\pi/2$).
  • Color: Three separate channels for Red, Green, and Blue.

The authors also created a specific "zero-shot" version of the dataset by removing certain combinations of factors (e.g., "red teapots seen from above") to test if models could generalize to entirely unseen scenarios.

Key Insights

  • Disentanglement $\neq$ Completeness: A model can be perfectly disentangled (each code variable only affects one factor) but still be "overcomplete" (one factor is spread across many code variables). Distinguishing between these two properties is crucial for understanding model behavior.
  • The Choice of Regressor Matters: If a Random Forest can predict a factor but a Lasso regressor cannot, it indicates the model has captured the information, but in a complex, non-linear way. This suggests the model hasn't truly "disentangled" the factor into a simple, usable form.
  • Topology Constraints: If a factor is circular (like azimuth/rotation) but the model uses a standard normal prior (which is "flat"), the model will likely struggle to represent that factor in a single, linear code variable. It will instead "tangle" the rotation across multiple variables (similar to sine and cosine representations).

Experimental Results

The authors compared several popular models, including PCA, VAE, $\beta$-VAE, and InfoGAN:

  • InfoGAN generally achieved the highest scores for both disentanglement and completeness when using the Lasso regressor, indicating that it learns very "explicit" and sparse representations.
  • VAEs and $\beta$-VAEs were highly informative (exhibiting low prediction error), but they often struggled with completeness, frequently spreading a single generative factor across multiple latent variables.
  • PCA performed the worst across all metrics, confirming that linear methods are insufficient for disentangling the complex, non-linear factors found in image data.
  • Zero-shot Generalization: Interestingly, while InfoGAN was better at disentangling, VAEs were slightly better at predicting factor values in the "zero-shot" gap (unseen combinations). This suggests a potential trade-off between extreme disentanglement and robust generalization.

Limitations and Future Work

The primary limitation of this framework is its reliance on ground-truth latent factors, which are typically only available for synthetic datasets. Evaluating disentanglement "in the wild" on real-world data where underlying factors are unknown remains an open challenge. Future work should explore applying this framework to other modalities beyond images and investigating how these quantitative metrics correlate with downstream task performance, such as in reinforcement learning or robotic manipulation.

Summary updated

Abstract

Recent AI research has emphasised the importance of learning disentangled representations of the explanatory factors behind data. Despite the growing interest in models which can learn such representations, visual inspection remains the standard evaluation metric. While various desiderata have been implied in recent definitions, it is currently unclear what exactly makes one disentangled representation better than another. In this work we propose a framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available. Three criteria are explicitly defined and quantified to elucidate the quality of learnt representations and thus compare models on an equal basis. To illustrate the appropriateness of the framework, we employ it to compare quantitatively the representations learned by recent state-of-the-art models.
BibTeX
@inproceedings{eastwood2018framework,
  author = {Cian Eastwood and Christopher K. I. Williams},
  title = {{A Framework for the Quantitative Evaluation of Disentangled Representations}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=By-7dz-AZ},
}
Figure