Curvature as a Source of Identifiability for Non-Commutative Factors

Learning "disentangled" representations means finding latent coordinates in which each real-world factor (an object's 3D orientation, the lighting direction, the object's color) can be read off and changed independently. A well-known theorem says this cannot be done from unlabeled, independently drawn images: many entangled latent codes explain the data exactly as well as the true one. The known escapes from this theorem either add weak supervision, such as paired images, or restrict the decoder (the network mapping latents to images) to be locally distance-preserving and require non-Gaussian factors. Both escapes assume each factor is a single scalar axis on a locally flat space. Factors such as 3D rotation fail both assumptions. They are multi-dimensional, their component motions do not commute, and the space they live on is curved.

This note argues that curvature, the property that breaks the existing guarantees, can supply identifiability instead. On a curved space, carrying a direction around a small loop and back (parallel transport) returns it rotated. The amount of rotation is called holonomy, and it has a clean consequence. By the de Rham decomposition theorem, a Riemannian manifold splits essentially uniquely into irreducible curved pieces plus one flat piece. Only the flat piece keeps the rotational ambiguity that the impossibility result exploits. We propose an autoencoder whose decoder is regularized to pull back a product metric onto groups of latent dimensions ("blocks"), not just a block-diagonal one. We prove and test that, under local isometry, curved non-commutative factors such as $\mathrm{SO}(3)$ pose are then identified as whole blocks from unlabeled images, with no non-Gaussianity requirement. Non-Gaussianity is still needed only for the flat remainder.

Core Claim

Assume three things: the image generator is a local isometry from a product of factor manifolds onto the image manifold, measured in pixel distance at small scales; the latent model is a local isometry onto a product-metric latent space; and both explain the same data. Then the learned blocks match the true factors up to permutation and a within-factor isometry for every factor with non-zero curvature. The only remaining ambiguity is an orthogonal mixing inside the union of flat factors, which non-Gaussian independence can resolve as in When Is Unsupervised Disentanglement Possible?. The measure-preserving entangling maps in Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations are generic homeomorphisms. Those that also preserve a product metric with curved factors form a small set, and that is where the identifiability comes from.

The method rests on a distinction that existing geometric regularizers miss. A block-diagonal pullback metric is not enough. A warped product, whose metric is $g = g_1(z_1) + \phi(z_1)^2 g_2(z_2)$ with a scalar warping function $\phi$, is block-diagonal, yet its holonomy mixes the blocks and its decomposition is not unique. The true product condition also requires each block's metric to depend only on that block's own coordinates. Writing $z=(z_1,\dots,z_K)$ for latent blocks, $J_k(z)$ for the decoder Jacobian restricted to block $k$, and $G_{kl}(z)=J_k^\top J_l$, the regularizer is

$$\mathcal{R}(z)=\sum_{k\neq l}\|G_{kl}(z)\|_F^2+\sum_{k\neq l}\big\|\partial_{z_l} G_{kk}(z)\big\|_F^2 + \lambda\,\big\|G(z)-\mathrm{blockdiag}(\bar G_1,\dots,\bar G_K)\big\|_{\text{loc}}^2 .$$

The first term enforces block-orthogonality. The second, the cross-block metric derivative, removes warping and is what makes holonomy block-preserving. The third is a local-isometry term in the style of Isometric Quotient Variational Auto-Encoders for Structure-Preserving Representation Learning. It is applied only along mini-batch neighborhoods, so that each block can carry its own curved geometry instead of being forced flat.

Relation to Prior Work

The motivating evidence comes from three papers. Towards a Definition of Disentangled Representations defines disentanglement as a decomposition of the representation into subspaces acted on by separate subgroups, and shows that non-commutative groups need multi-dimensional subspaces. It offers no learning algorithm. Disentangling by Subspace Diffusion (GEOMANCER) applies de Rham decomposition directly. It estimates tangent spaces by local principal component analysis and builds a connection Laplacian, a graph operator that diffuses tangent vectors across the data. With the true latent metric it recovers $\mathrm{SO}(3)$ and lighting subspaces on Stanford 3D Objects with about 0.02 radians of subspace angle error. However, it is non-parametric, needs that metric oracle, cannot encode new samples, and states no identifiability theorem. Our proposal turns its geometric principle into a trainable encoder-decoder and supplies the missing theorem. Holonomy Grid Codes for Generalisation Under Directed Actions proves that non-trivial holonomy forces representations of at least $q$ dimensions. That supports our premise that curvature dictates block size, but the paper is restricted to twisted Abelian grids and contains no learning.

The constraining work is the identifiability literature. When Is Unsupervised Disentanglement Possible? proves identifiability under local isometry plus non-Gaussianity using HLLE+ICA (Hessian locally linear embedding followed by independent component analysis) and reaches a mean correlation coefficient near 0.97 on GAN manifolds. Its argument requires local flatness, which is exactly what our curved factors violate. Geometric Inductive Biases for Identifiable Unsupervised Learning of Disentangled Representations (GDRAE) orthogonalizes the decoder Jacobian over scalar axes and assumes flat parallel transport. Our block-orthogonality term generalizes its Jacobian constraint, and our cross-derivative term handles the zero-holonomy assumption GDRAE makes. Desiderata for Representation Learning from Identifiability, Disentanglement, and Group-Structuredness warns that identifiability alone does not guarantee correct topology. We therefore give each block an embedding chart with more coordinates than its intrinsic dimension, so that $\mathrm{SO}(3)$ need not be torn into a box.

Several methods could be reused or extended. Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations (ISA-VAE) breaks rotational ambiguity using a non-spherical prior over subspaces. We use it only for the flat remainder. Its identification is statistical, whereas ours is geometric. Commutative Lie Group VAE for Disentanglement Learning explicitly penalizes non-commutation of its generators, so it is structurally unable to represent $\mathrm{SO}(3)$; it serves as our commutative control. Learning disentangled representations via product manifold projection also uses a product-of-submanifolds latent space, but it relies on paired images and fixes block dimensions by hand. It is our weakly supervised reference. Disentangled representation learning through unsupervised symmetry group discovery discovers block structure from labeled actions, which we do not have.

Research Program

The theory component states the identifiability result for a local de Rham decomposition. We work locally, or on the universal cover, because $\mathrm{SO}(3)$ is not simply connected. We separate two claims: uniqueness of the curved factors, which holds without distributional assumptions, and uniqueness of the flat factor, which needs non-Gaussian independent subspaces. We will also state the failure case honestly. When the true curvature of a factor is small relative to sampling noise, the guarantee degrades continuously toward the flat-case ambiguity. The theory should predict how sample size trades off against curvature.

The model is a deterministic autoencoder with an over-complete latent space. Every few thousand steps, block membership is re-estimated. We build a coupling graph whose edge weight between latent dimensions $a$ and $b$ is the mini-batch average of $|G_{ab}|$ plus $|\partial_b G_{aa}|$, and we take its connected components. Block dimensionality is thus discovered rather than fixed, which mirrors GEOMANCER's use of the connection Laplacian spectrum to count factors. The regularizer is estimated with forward-mode Jacobian-vector products along random within-block probe directions. The metric derivative uses a nested Jacobian-vector product, so the cost is a small constant number of extra forward passes per probe and no full Jacobian is ever formed. We will report the training overhead. For a comparable geometric regularizer, Disentangled Representation Learning with the Gromov-Monge Gap reports roughly 3 ms per batch on 128×128 Shapes3D, which gives a reference for what counts as acceptable.

Baselines

The naive reference point is a plain autoencoder and a β-VAE with the same encoder, decoder, and latent width. GEOMANCER reports that this class of model often fails to beat random chance on subspace angle error on Stanford 3D Objects. Our method must clear that bar by a wide margin on every curved-factor dataset.

The strongest unsupervised competitor on flat factors is HLLE+ICA and its distilled Hessian Auto-Encoder from When Is Unsupervised Disentanglement Possible?, which reports a mean correlation coefficient near 0.97 on the 10 StyleGAN2/GANSpace manifolds. We run its code on the same samples. We also run a variant that replaces ICA with independent subspace analysis, so that it can output blocks. On the GAN manifolds, which are flat, we only need to match it within noise. On curved-factor data, beating it means a lower subspace angle error and a higher block-wise canonical correlation with the true factors across five seeds.

The strongest method on non-commutative factors is GEOMANCER. With the true metric it reaches about 0.02 radians on Stanford 3D Objects. Since that figure uses an oracle we do not have, we run GEOMANCER two ways. The first follows its intended input: the true latent metric. The second is fed pixel distances and, separately, the latent codes of a pretrained autoencoder, with an equal neighbor-count tuning budget. Beating it means outperforming the non-oracle versions and coming within a small factor of the oracle number, while also encoding held-out images.

ISA-VAE and the Commutative Lie Group VAE are run with their published priors and penalties, with the same architecture and hyperparameter search budget as ours. The file reports that ISA-VAE achieves the best MIG on 3D Faces, but it gives no numeric value. The Commutative Lie Group VAE reports a FactorVAE metric of 86.1 on dSprites, which we use as a sanity check on that commutative benchmark.

As a weakly supervised upper reference, the product manifold projection model reaches a BetaVAE score of 99.2% on SmallNORB with paired data, against 87.9% for Ada-GVAE. We do not expect to match it without pairs. Instead we measure how much of the gap between Ada-GVAE and that model our unsupervised method closes.

The key ablation removes the cross-block metric-derivative term and keeps block-orthogonality and local isometry. This is a block-wise GDRAE/IQVAE hybrid. If the theory is right, this ablation should succeed on flat and commutative data and fail on warped and non-commutative data.

Why not just…?

Why not just run ISA-VAE, which already finds multi-dimensional subspaces without labels? ISA-VAE identifies subspaces through the shape of the prior. That only rules out linear rotations of the latent space, not non-linear entangling maps, and its theory says nothing about curved factor spaces. Our curvature-sweep experiment speaks to this directly. If ISA-VAE recovers $\mathrm{SO}(3)$ blocks as well as our model when pose is distributed uniformly over the group (a distribution whose radial profile gives a non-spherical prior little to grip), the geometric term is unnecessary and we would say so.

Why not just apply GEOMANCER to a pretrained autoencoder's latent codes, as CITRIS-NF does with flows on a frozen space? This is a real threat because it reuses a validated algorithm. Its weakness is that a frozen autoencoder's latent metric is arbitrary. The impossibility theorem says the autoencoder may already have warped the factors together, and GEOMANCER then needs exactly the metric oracle that is missing. We include this two-stage pipeline as a baseline. If it matches our end-to-end model, the contribution reduces to the identifiability theorem, and we would report that.

Why not just use HLLE followed by independent subspace analysis? HLLE recovers coordinates up to an orthogonal map only when the manifold is locally isometric to flat space. On $S^2$ or $\mathrm{SO}(3)$ its Hessian null space does not exist in the assumed form. We therefore expect it to degrade as curvature grows, and the curvature sweep is designed to show where this happens. On flat data it should remain as strong as reported, and we do not claim otherwise.

Why not just enforce block-diagonal Jacobians, generalizing GDRAE? Warped products satisfy block-diagonality while entangling the factors through holonomy. The ablation above tests whether the extra derivative term matters in practice.

Proposed Experiments

The decisive experiment is a controlled curvature sweep. We render a single object (from the Stanford 3D Objects setup used by GEOMANCER) under three factor families. The first is a flat torus $T^2$ of in-plane translations, sampled from a Gaussian-like and from a uniform distribution. The second is a lighting direction on $S^2$ whose sphere radius, and hence curvature, is varied. The third is full $\mathrm{SO}(3)$ pose sampled uniformly. The theory makes three predictions. Curved factors should be recovered as blocks with low subspace angle error regardless of their sampling distribution. The flat factor should be recovered only when its distribution is non-Gaussian. Recovery error should grow as curvature shrinks toward zero at fixed sample size. If the curved factors are not recovered under Gaussian-like or uniform sampling, the central claim is false.

The second set of experiments uses standard benchmarks with partly curved factors. These are SmallNORB (azimuth and elevation together form a sphere-like pose factor), Cars3D, MPI3D (two arm rotation degrees of freedom on real images), and Temporal Causal3DIdent frames treated as i.i.d. images, comparing against CITRIS-NF's reported $R^2$ of 0.98, which it obtains using temporal interventions we do not use. Metrics are subspace angle error where true tangent spaces are available, block-wise canonical correlation, the multi-dimensional MIG-KM metric from the product-manifold paper, DCI, and reconstruction error. For the β-TCVAE + GMG model, Disentangled Representation Learning with the Gromov-Monge Gap reports DCI-D of 96.9 on Shapes3D, which serves as a flat-factor control. On those flat-factor controls, which are Shapes3D and the GAN manifolds, the method must not lose to its flat-space competitors. A curvature-aware method that hurts the flat case would be a poor trade.

A final check tests whether block dimensions are discovered correctly. For each dataset we report the number and size of the coupling-graph components the method finds, against the ground-truth factor dimensions (3 for $\mathrm{SO}(3)$, 2 for $S^2$). We also report the result of deliberately starting with too many and too few latent dimensions. This addresses the open problem of hand-fixed subspace dimensionality.