Abstract
To evaluate disentangled representations several metrics have been proposed. However, theoretical guarantees for conventional metrics of disentanglement are missing. Moreover, conventional metrics do not have a consistent correlation with the outcomes of qualitative studies. In this paper we analyze metrics of disentanglement and their properties. We conclude that existing metrics of disentanglement were created to reflect different characteristics of disentanglement and do not satisfy two basic desirable properties: (1) assign a high score to representations that are disentangled according to the definition; and (2) assign a low score to representations that are entangled according to the definition. In addition, we propose a new metric of disentanglement and prove that it satisfies both of the properties.
The Disentanglement Problem
In representation learning, the objective is to learn a "disentangled" representation where individual dimensions of the latent code correspond to independent, interpretable generative factors of the data. For instance, in a dataset of rectangles, one latent dimension might represent 'width' while another represents 'length'.
While algorithms like $\beta$-VAE or FactorVAE are designed for this, measuring their success has proven difficult. Current metrics—such as BetaVAE score, FactorVAE score, DCI, SAP, and MIG—often yield conflicting results and fail to align with human intuition. This research addresses the lack of theoretical grounding for these metrics, asking whether a high score truly guarantees disentanglement.
Defining a Good Metric
The authors propose that any valid disentanglement metric must satisfy two fundamental properties:
- Property 1: It must assign a high score to representations that are disentangled by definition.
- Property 2: It must assign a low score to representations that are entangled by definition.
Disentanglement itself is characterized by two directions of influence:
- Characteristic 1: A change in one latent dimension affects only one generative factor.
- Characteristic 2: A change in one generative factor affects only one latent dimension.
Why Existing Metrics Fail
Through mathematical "stress tests," the authors reveal significant flaws in popular benchmarks:
- BetaVAE and FactorVAE: These metrics are overly "optimistic." Because they use a classifier to identify which factor was held constant, they can award a perfect score even if information about that factor is leaked across multiple entangled latent dimensions.
- SAP (Separated Attribute Predictability): This metric relies on linear regression ($R^2$). It fails to recognize perfect disentanglement if the relationship between a factor and a latent is non-linear (e.g., $c = z^{15}$), leading to unfairly low scores.
- MIG (Mutual Information Gap): While MIG is a strong predecessor—satisfying Property 1—it still falters on Property 2 in specific edge cases.
3CharM: A Theoretically Sound Alternative
To resolve these issues, the authors introduce 3CharM (Three Characteristics Metric). The algorithm follows a rigorous path:
- Informativeness: Construct a Mutual Information (MI) matrix between all latent variables and generative factors.
- Hypothesis Generation: For each latent variable, determine its "primary" generative factor based on the highest MI.
- Scoring: Measure how much more information a latent variable contains about its primary factor compared to others.
- Selection & Aggregation: For each generative factor, select the single best-performing latent variable and average these scores.
By explicitly enforcing a one-to-one mapping, 3CharM penalizes models that spread information about a single factor across many dimensions.
Results and Comparison
The theoretical "Report Card" highlights the superiority of the proposed approach:
| Metric | Property 1 (High for Good) | Property 2 (Low for Bad) |
|---|---|---|
| BetaVAE | No | No |
| FactorVAE | No | No |
| DCI | No | No |
| SAP | No | No |
| MIG | Yes | Yes |
| 3CharM (Ours) | Yes | Yes |
Empirical validation on standard datasets like dSprites and Cars3D confirms that while metrics may correlate on simple tasks, 3CharM provides a much more reliable signal for complex or intentionally "tricky" representations.
Conclusion
The field of unsupervised representation learning has long relied on inconsistent metrics. By formalizing the requirements for a good metric and proving the failures of existing ones, this work provides a path toward more rigorous evaluation. While 3CharM faces challenges common to MI-based methods—such as estimation difficulty from finite samples—it stands as the first metric with the theoretical guarantees necessary to truly measure disentanglement.
Summary updated