How to Not Measure Disentanglement

Anna Sepliarskaia, Julia Kiseleva, Maarten de Rijke

Abstract

To evaluate disentangled representations several metrics have been proposed. However, theoretical guarantees for conventional metrics of disentanglement are missing. Moreover, conventional metrics do not have a consistent correlation with the outcomes of qualitative studies. In this paper we analyze metrics of disentanglement and their properties. We conclude that existing metrics of disentanglement were created to reflect different characteristics of disentanglement and do not satisfy two basic desirable properties: (1) assign a high score to representations that are disentangled according to the definition; and (2) assign a low score to representations that are entangled according to the definition. In addition, we propose a new metric of disentanglement and prove that it satisfies both of the properties.

The Disentanglement Problem

In representation learning, the objective is to learn a "disentangled" representation where individual dimensions of the latent code correspond to independent, interpretable generative factors of the data. For instance, in a dataset of rectangles, one latent dimension might represent 'width' while another represents 'length'.

Disentangled vs. entangled representations of a generative process (e.g., creating a rectangle from length and width).

While algorithms like $\beta$-VAE or FactorVAE are designed for this, measuring their success has proven difficult. Current metrics—such as BetaVAE score, FactorVAE score, DCI, SAP, and MIG—often yield conflicting results and fail to align with human intuition. This research addresses the lack of theoretical grounding for these metrics, asking whether a high score truly guarantees disentanglement.

Defining a Good Metric

The authors propose that any valid disentanglement metric must satisfy two fundamental properties:

  1. Property 1: It must assign a high score to representations that are disentangled by definition.
  2. Property 2: It must assign a low score to representations that are entangled by definition.

Disentanglement itself is characterized by two directions of influence:

  • Characteristic 1: A change in one latent dimension affects only one generative factor.
  • Characteristic 2: A change in one generative factor affects only one latent dimension.

Why Existing Metrics Fail

Through mathematical "stress tests," the authors reveal significant flaws in popular benchmarks:

  • BetaVAE and FactorVAE: These metrics are overly "optimistic." Because they use a classifier to identify which factor was held constant, they can award a perfect score even if information about that factor is leaked across multiple entangled latent dimensions.
  • SAP (Separated Attribute Predictability): This metric relies on linear regression ($R^2$). It fails to recognize perfect disentanglement if the relationship between a factor and a latent is non-linear (e.g., $c = z^{15}$), leading to unfairly low scores.
  • MIG (Mutual Information Gap): While MIG is a strong predecessor—satisfying Property 1—it still falters on Property 2 in specific edge cases.
Illustration of complex mappings between generative factors ($z$) and latent factors ($c$), highlighting the difficulty in identifying one-to-one correspondences.

3CharM: A Theoretically Sound Alternative

To resolve these issues, the authors introduce 3CharM (Three Characteristics Metric). The algorithm follows a rigorous path:

  1. Informativeness: Construct a Mutual Information (MI) matrix between all latent variables and generative factors.
  2. Hypothesis Generation: For each latent variable, determine its "primary" generative factor based on the highest MI.
  3. Scoring: Measure how much more information a latent variable contains about its primary factor compared to others.
  4. Selection & Aggregation: For each generative factor, select the single best-performing latent variable and average these scores.

By explicitly enforcing a one-to-one mapping, 3CharM penalizes models that spread information about a single factor across many dimensions.

Results and Comparison

The theoretical "Report Card" highlights the superiority of the proposed approach:

Metric Property 1 (High for Good) Property 2 (Low for Bad)
BetaVAE No No
FactorVAE No No
DCI No No
SAP No No
MIG Yes Yes
3CharM (Ours) Yes Yes

Empirical validation on standard datasets like dSprites and Cars3D confirms that while metrics may correlate on simple tasks, 3CharM provides a much more reliable signal for complex or intentionally "tricky" representations.

Conclusion

The field of unsupervised representation learning has long relied on inconsistent metrics. By formalizing the requirements for a good metric and proving the failures of existing ones, this work provides a path toward more rigorous evaluation. While 3CharM faces challenges common to MI-based methods—such as estimation difficulty from finite samples—it stands as the first metric with the theoretical guarantees necessary to truly measure disentanglement.

Summary updated

Abstract

To evaluate disentangled representations several metrics have been proposed. However, theoretical guarantees for conventional metrics of disentanglement are missing. Moreover, conventional metrics do not have a consistent correlation with the outcomes of qualitative studies. In this paper we analyze metrics of disentanglement and their properties. We conclude that existing metrics of disentanglement were created to reflect different characteristics of disentanglement and do not satisfy two basic desirable properties: (1) assign a high score to representations that are disentangled according to the definition; and (2) assign a low score to representations that are entangled according to the definition. In addition, we propose a new metric of disentanglement and prove that it satisfies both of the properties.
BibTeX
@misc{sepliarskaia2019how,
  author = {Anna Sepliarskaia and Julia Kiseleva and Maarten de Rijke},
  title = {{How to Not Measure Disentanglement}},
  year = {2019},
  eprint = {1910.05587},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.05587},
}
Figure