Evaluating Disentangled Representations

Anna Sepliarskaia, Julia Kiseleva, Maarten de Rijke

Abstract

To evaluate disentangled representations several metrics have been proposed. However, theoretical guarantees for conventional metrics of disentanglement are missing. Moreover, conventional metrics do not have a consistent correlation with the outcomes of qualitative studies. In this paper we analyze metrics of disentanglement and their properties. We conclude that existing metrics of disentanglement were created to reflect different characteristics of disentanglement and do not satisfy two basic desirable properties: (1) assign a high score to representations that are disentangled according to the definition; and (2) assign a low score to representations that are entangled according to the definition. In addition, we propose a new metric of disentanglement and prove that it satisfies both of the properties.

The Challenge of Measuring Disentanglement

In representation learning, the objective is often to derive a "disentangled" representation—one where individual dimensions of the data (e.g., the width of a rectangle or the rotation of an object) are captured by independent latent variables. While intuitively appealing, disentanglement has historically lacked a single, standard mathematical definition.

Comparison between disentangled and entangled representations of a rectangle's dimensions.

Researchers have proposed various metrics to evaluate model performance, such as BetaVAE, FactorVAE, Mutual Information Gap (MIG), and Separated Attribute Predictability (SAP). However, these metrics frequently disagree with one another and with qualitative human judgment. This research addresses a fundamental question: Do current metrics actually measure what they claim to measure? The authors evaluate whether these metrics satisfy two basic properties: assigning a high score to truly disentangled representations (Property 1) and assigning a low score to entangled ones (Property 2).

Defining Disentanglement Characteristics

The paper categorizes disentanglement into two distinct characteristics:

  • Characteristic 1: A change in one latent dimension corresponds to a change in exactly one generative factor (and nothing else).
  • Characteristic 2: A change in one generative factor leads to a change in exactly one latent dimension.

Mathematical proofs demonstrate that most existing metrics (BetaVAE, FactorVAE, DCI, SAP) fail to satisfy both Property 1 and Property 2. Even the MIG metric, which performs better than others, only reflects Characteristic 2.

To bridge this gap, the authors propose 3CharM (Three Characteristic Metric). The 3CharM algorithm operates in three stages:

  1. Hypothesis Generation: For every latent variable, it identifies the generative factor it reflects most.
  2. Selection: For every generative factor, it selects the "best" corresponding latent variable from the candidates.
  3. Scoring: It calculates a final score based on the mutual information between these pairs, normalized by entropy.

Experimental Setup and Datasets

The evaluation utilizes two standard benchmark datasets where generative factors are known, allowing for precise ground-truth calculations:

  • dSprites: 2D shapes varying in scale, rotation, and position.
  • Cars3D: 3D rendered cars varying in elevation and azimuth.

Key Insights and Analysis of Existing Metrics

The "One-to-Many" Trap

A significant finding is that many metrics are overly "optimistic." For instance, a representation can satisfy Characteristic 1 (one latent maps to one factor) but still be entangled if multiple latents capture the same factor. Most existing metrics fail to penalize this redundancy.

Illustration of a one-to-many mapping where a single generative factor influences multiple latent variables, a common failure mode for standard metrics.

Linearity Bias and Metric Performance

Metrics like the SAP score rely on linear regression. If the relationship between a latent variable and a factor is non-linear (e.g., $c = z^{15}$), SAP will incorrectly report a low disentanglement score even if the mapping is perfectly one-to-one.

The paper’s most striking result is summarized in a comparative analysis showing that BetaVAE, FactorVAE, DCI, and SAP fail to satisfy both Property 1 and Property 2. 3CharM is identified as the only metric that satisfies both properties for a comprehensive definition of disentanglement by requiring a strict one-to-one mapping in both directions.

Conclusion

While 3CharM provides a more robust theoretical framework, practical challenges remain. Calculating mutual information is more computationally intensive than the simple classifiers used by BetaVAE. Furthermore, the reliance on synthetic datasets with known generative factors highlights the ongoing challenge of applying these rigorous metrics to complex, real-world data where ground truth is unknown.

Summary updated

Abstract

To evaluate disentangled representations several metrics have been proposed. However, theoretical guarantees for conventional metrics of disentanglement are missing. Moreover, conventional metrics do not have a consistent correlation with the outcomes of qualitative studies. In this paper we analyze metrics of disentanglement and their properties. We conclude that existing metrics of disentanglement were created to reflect different characteristics of disentanglement and do not satisfy two basic desirable properties: (1) assign a high score to representations that are disentangled according to the definition; and (2) assign a low score to representations that are entangled according to the definition. In addition, we propose a new metric of disentanglement and prove that it satisfies both of the properties.
BibTeX
@misc{sepliarskaia2019evaluating,
  author = {Anna Sepliarskaia and Julia Kiseleva and Maarten de Rijke},
  title = {{Evaluating Disentangled Representations}},
  year = {2019},
  eprint = {1910.05587},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.05587},
}
Figure