Measuring Orthogonality in Representations of Generative Models

Robin C. Geyer, Alessandro Torcinovich, João B. S. Carvalho, Alexander Meyer, Joachim M. Buhmann

Abstract

Traditional disentanglement metrics are often too restrictive, penalizing high-quality representations simply because they are not aligned with the latent space's axes. This paper introduces Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR), new metrics that measure the independence of generative factors through subspace orthogonality rather than axis alignment. We demonstrate that these metrics are invariant to rotation and correlate significantly more strongly with downstream task performance than established benchmarks like MIG or DCI.

Introduction: The Axis-Alignment Trap

In unsupervised representation learning, the goal is to compress high-dimensional data into a low-dimensional "code" that captures essential generative factors (e.g., color, shape, scale). Historically, the community has defined "disentanglement" as a state where each generative factor is captured by exactly one dimension in the latent space.

However, this definition imposes an unnecessary geometric constraint. As shown in the figure below, a representation can be perfectly clear and independent even if it is rotated relative to the axes. Existing metrics like DCI or MIG would fail such a model, despite its high utility for downstream tasks. This creates a gap where we penalize useful representations for failing a specific, arbitrary geometric orientation.

Visualizing factor alignment: (i) axis-aligned, (ii) rotated but orthogonal, and (iii) non-orthogonal representations.

Methodology: Importance-Weighted Metrics

To address this, we propose a shift from "disentanglement" to "orthogonality." Our framework consists of two primary components:

1. Generative Component Analysis (GCA)

GCA is a novel procedure used to identify the linear subspace where a specific generative factor resides. It employs a Linear Neural Network (LNN) "spine" to progressively project the latent code into smaller dimensions. Simultaneously, non-linear regressor "heads" ensure that the factor's information is preserved throughout the projection. This process identifies which directions in the latent space are truly relevant for a specific factor, yielding "importance weights" ($\alpha$) for each dimension.

The GCA framework uses a linear spine and non-linear heads to identify factor subspaces and compute importance weights.

2. IWO and IWR Metrics

  • Importance-Weighted Orthogonality (IWO): This measures the geometric separation between the subspaces of different factors. By using the importance weights derived from GCA, IWO ensures that dimensions carrying more information about a factor are weighted more heavily when calculating perpendicularity.
  • Importance-Weighted Rank (IWR): This measures the concentration of a factor's information. A high IWR indicates that a factor is distilled into a few key dimensions, whereas a low IWR suggests the information is spread thinly across the space.

Experimental Validation

We validated these metrics using 2,160 representations trained on six Variational Autoencoder (VAE) variants across standard benchmarks (dSprites, Cars3D, Shapes3D, etc.) and synthetic datasets designed to test rotation invariance.

Key Results

  • Robustness to Rotation: In experiments using Random Orthogonal Projections (ROP), traditional Disentanglement (D) and Completeness (C) scores collapsed (e.g., from 0.99 to 0.05). In contrast, IWO and IWR remained nearly constant, proving they measure intrinsic representation quality rather than orientation.
  • Predictive Power for Utility: Across all datasets, IWO and IWR showed significantly higher correlations with downstream task performance (such as regression and segmentation) than MIG or DCI. On the Cars3D dataset, while traditional metrics often showed negative correlations with performance, IWO/IWR remained strong positive predictors.
  • Qualitative Alignment: Latent space traversals confirmed that GCA correctly identifies dimensions varying a single factor, even when those dimensions appear "entangled" under axis-aligned frameworks.

Conclusion

By moving beyond the rigid requirement of axis-alignment, IWO and IWR provide a more accurate measure of a representation's actual utility. While these metrics currently require ground-truth supervision for evaluation, they offer a robust tool for researchers to assess the intrinsic independence of learned generative factors. Future work may involve integrating these orthogonality principles directly into the training objectives of generative models to encourage the learning of more robust, usable representations by design.

Summary updated

BibTeX
@article{geyer2024measuring,
  author = {Robin C. Geyer and Alessandro Torcinovich and João B. S. Carvalho and Alexander Meyer and Joachim M. Buhmann},
  title = {{Measuring Orthogonality in Representations of Generative Models}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=TSUprKRga1},
}
Figure