Neural Isometries: Taming Transformations for Equivariant ML

Thomas W. Mitchel, Michael J. Taylor, Vincent Sitzmann

Introduction: The Challenge of Real-World Equivariance

Equivariance—the property where a model's internal representations transform predictably with its input—is a cornerstone of robust generalization in machine learning. While symmetries like translation and rotation are well-handled by standard architectures, real-world data often undergoes complex, non-linear transformations. For instance, a 3D camera movement translates to intricate optical flow and perspective shifts in 2D image space. Traditional Geometric Deep Learning often requires these transformations to belong to known mathematical groups, necessitating handcrafted architectures. Neural Isometries (NIso) aims to bridge this gap by learning a latent space where these complex transformations become tractable, linear, and highly structured.

Neural Isometries (NIso) Framework

NIso is an autoencoder framework that enforces a specific geometric structure on its latent space to tame complex transformations. Instead of treating transformations as black-box neural layers, NIso represents them as Functional Maps ($\tau$)—linear operators that relate the encoding of one observation to another.

The NIso framework maps complex input transformations into structured isometries within a latent space.

The framework relies on several key technical innovations:

  1. Isometric Regularization: The latent maps are constrained to be isometries, meaning they must preserve a learned inner product and commute with a learned functional operator $\Omega$ (analogous to the Laplacian in physics).
  2. Spectral Decomposition: The operator $\Omega$ is decomposed into eigenvalues and eigenfunctions. By forcing transformation maps to be diagonal in this "eigen-basis," the network is pressured to disentangle the content of the data from the transformation acting upon it.
  3. Multiplicity Loss: A novel loss function encourages the learned operator to have distinct eigenvalues, which effectively sparsifies the transformation maps, making them easier for downstream networks to process.

Key Insights: Learning the "Physics" of Data

A core strength of NIso is its ability to discover the "natural frequencies" of data. In experiments with Toric and Spherical data, the model recovered operators that structurally matched ground-truth mathematical Laplacians. This suggests that NIso effectively learns the underlying geometry of the data manifold. Furthermore, the learned latent maps ($\tau$) are so geometrically informative that camera poses can be regressed directly from them with high accuracy, outperforming standard representation learners like DINOv2.

Quantitative Results

The authors evaluated NIso across three distinct domains: homography-perturbed MNIST, 3D shape classification (SHREC ’11), and real-world 3D video sequences (CO3Dv2).

Quantitative performance and visual examples on the Homography-Perturbed MNIST dataset.

As shown in the results above, NIso achieved 92.52% accuracy on homography-perturbed MNIST, significantly outperforming the previous unsupervised state-of-the-art method, NFT (41.93%). Other key results include:

  • 3D Shape Mastery: On SHREC '11, NIso reached 90.26% accuracy, outperforming specialized Möbius Convolution architectures (86.5%) designed specifically for conformal 3D data.
  • Robust Pose Estimation: On the CO3Dv2 dataset, NIso significantly outperformed Transformer-based baselines in camera pose estimation, maintaining low error even when predicting poses across large frame gaps.

Limitations and Future Work

Currently, the learned operator $\Omega$ is tied to the specific domain it was trained on, making transferability between different data types a challenge. Additionally, the model does not yet explicitly handle occlusions or partial views, which are common in real-world 3D vision. Future work will focus on scaling this approach to complex non-rigid motion and full-scale autonomous driving scenes.

Summary updated

BibTeX
@inproceedings{mitchel2024neural,
  author = {Thomas W. Mitchel and Michael J. Taylor and Vincent Sitzmann},
  title = {{Neural Isometries: Taming Transformations for Equivariant ML}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=kCabCEhQWv},
}
Figure