Modern machine learning often assumes data lives in a flat, Euclidean world. However, many real-world datasets—from brain imaging tensors to document embeddings—naturally reside on curved surfaces or manifolds. Applying standard regularization techniques to such non-Euclidean data can lead to severe performance degradation or training failures.
This paper introduces Geometrically Regularized Autoencoders (GRAEs), a framework that generalizes denoising and contractive autoencoders to respect the intrinsic geometry of non-Euclidean domains. By doing so, these models achieve more stable training, better density estimation, and superior downstream performance.
The Problem with Flat Assumptions
Standard autoencoders rely on regularization to learn robust representations. Denoising Autoencoders (DAEs) add noise, while Contractive Autoencoders (CAEs) penalize the Jacobian. Unfortunately, these methods are typically "coordinate-dependent."
If your data lives on a sphere, adding standard Euclidean noise might push a data point off the surface entirely. Furthermore, using specific coordinate systems (like latitude and longitude) causes standard regularization to behave inconsistently—differently near the poles than at the equator. This lack of coordinate invariance can create singularities, destabilizing training and preventing the model from learning the true underlying data distribution.
Geometric Generalizations
To address this mismatch, the authors propose two manifold-aware generalizations:
- Geometric Denoising Autoencoder (GDAE): Instead of adding simple Gaussian noise, GDAE corrupts data by sampling a vector in the tangent space of the manifold and using the exponential map to move the point along a geodesic (the shortest path on a curved surface). The reconstruction loss is then calculated using geodesic distance rather than the standard $L_2$ norm.
- Geometric Reconstruction Contractive Autoencoder (GRCAE): Generalizing the CAE, this method uses the theory of harmonic maps to penalize the "Dirichlet energy" of the reconstruction mapping. This ensures the regularization is intrinsic to the manifold's geometry and consistent regardless of parameterization.
A key theoretical contribution is proving that these models can estimate the "geometric score"—the gradient of the log-probability density on the manifold. This enables their use as generative models via Riemannian Langevin Monte Carlo sampling.
Key Results Across Domains
The authors evaluated GRAEs across several non-Euclidean domains, including hyperspheres, Symmetric Positive-Definite (SPD) matrices, and statistical manifolds.
- Score Estimation: On SPD matrix manifolds, GRCAE significantly outperformed standard Euclidean baselines and existing log-density gradient estimators.
- Document Clustering: Using Newsgroup20 document embeddings projected onto a 50-dimensional hypersphere, GDAE achieved an Adjusted Rand Index (ARI) of 0.465, more than doubling the performance of a standard DAE (0.205).
- Medical Imaging (DTI): In Diffusion Tensor Imaging filtering, GDAE successfully cleaned noisy brain scans while strictly preserving the positive-definite constraint of the tensors. Standard Euclidean denoising often violates this constraint, yielding physically impossible results.
- Representation Learning: GDAE-based features proved more robust for transfer learning on point clouds, maintaining higher classification accuracy under input noise compared to vanilla autoencoders.
Limitations and Future Directions
While geometrically principled, these methods come with a computational cost. Calculating Riemannian metrics, exponential maps, and Jacobians involves heavy matrix operations. The authors note that GRCAE can be 30x to 50x slower to train than a standard DAE. Additionally, the current framework assumes the Riemannian metric of the data space is known a priori. Future work will need to focus on scaling these geometric operations for extremely high-dimensional manifolds and massive datasets.
Summary updated