Most machine learning models operate under a "flat earth" assumption, treating data as points in Euclidean space. However, many real-world datasets possess intrinsic structures that are fundamentally curved. Hierarchical data, such as language taxonomies or phylogenetic trees, naturally fit into hyperbolic spaces. Periodic data, like robot joint angles or molecular rotations, live on spheres or tori. Covariance matrices, essential in medical imaging and finance, inhabit the manifold of Symmetric Positive Definite (SPD) matrices.
When we force these "curved" concepts into "flat" latent spaces, models struggle with "manifold mismatch." A standard Variational Autoencoder (VAE) might assign probability mass to impossible coordinates or fail to capture the hierarchical nature of the data. To solve this, researchers are developing a toolkit for performing statistical estimation and generative modeling directly on Riemannian manifolds.
The Tangent Space and the Exponential Map
The core challenge of non-Euclidean modeling is that standard probability distributions, like the Gaussian, do not have a straightforward definition on a curved surface. You cannot simply "add" noise to a point on a sphere; the result would likely leave the surface entirely.
The solution, popularized by researchers like Anton Mallasto, Søren Hauberg, and Aasa Feragen in Probabilistic Riemannian submanifold learning with wrapped Gaussian process latent variable models, is to use the tangent space.
At any point $\mu$ on a manifold $M$, there exists a flat, Euclidean tangent space $T_\mu M$. We can define a standard Gaussian distribution within this flat space and then "wrap" it onto the manifold using a mathematical tool called the exponential map ($\exp_\mu$). This map takes a vector from the tangent space and follows a "straight line" (a geodesic) along the curved surface to find its new location.
This "wrapped" distribution allows us to:
- Sample by drawing from a normal distribution in the flat tangent space.
- Project those samples onto the manifold using the exponential map.
- Transport the distribution across the manifold using parallel transport, which moves the tangent space while preserving the orientation of the distribution.
As Emmanuel Chevallier and David B. Dunson demonstrate in Exponential-wrapped distributions on symmetric spaces, this framework makes statistical modeling on complex shapes like Grassmannians and hyperbolic spaces computationally tractable.
Generative Modeling in Curved Latent Spaces
Once we can define distributions on manifolds, we can rebuild generative architectures. The Poincaré Variational Auto-Encoder ($\mathcal{P}$-VAE), introduced by Emile Mathieu et al. in Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders, replaces the standard flat latent space with a Poincaré ball. Because the distance between points in hyperbolic space grows exponentially as you move toward the edge, it can embed tree-like structures with much lower distortion than Euclidean space.
To make these models trainable, the distributions must be differentiable. Yoshihiro Nagano and Shoichiro Yamaguchi in A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning developed the reparameterization trick for wrapped distributions. This allows the model to backpropagate through the exponential map, enabling end-to-end training of hyperbolic VAEs.
Beyond VAEs, researchers are extending other generative paradigms:
- Normalizing Flows: Heli Ben-Hamu and Maximilian Nickel developed Matching Normalizing Flows and Probability Paths on Manifolds, which learns vector fields on manifolds to transform simple priors into complex target densities.
- Diffusion Models: Luis A. Pérez Rey introduced Diffusion Variational Autoencoders, which use Brownian motion on closed manifolds (like spheres or tori) to define the latent diffusion process, naturally matching the data's topology.
Accounting for Curvature and Volume Distortion
A critical nuance in this field is that "wrapping" a distribution changes its density. When you project a flat distribution onto a curved surface, some areas are stretched and others are compressed. This is known as volume distortion.
To calculate the correct probability density on the manifold, models must include a Jacobian correction term. Kenji Fukumizu explored this in Statistical Estimation in Singular Models, noting that standard statistical tests often fail because they ignore the geometry of "singularities" or non-convex tangent cones.
More recently, Hanlin Yu and Arto Klami proposed the Riemannian Laplace Approximation with the Fisher Metric. Instead of approximating a posterior as a flat Gaussian, they use the Fisher Information Matrix as a Riemannian metric. This ensures the approximation is "intrinsic" to the probability space, leading to more robust uncertainty quantification in Bayesian Neural Networks (Federico Bergamin et al., Riemannian Laplace approximations for Bayesian neural networks).
Why It Works: The Power of Inductive Bias
The success of these methods stems from a better inductive bias. By choosing a manifold that matches the data's structure, the model doesn't have to "learn" the constraints; they are baked into the geometry.
In robotics, Noémie Jaquier and Leonel Dario Rozo show in Bringing robotics taxonomies to continuous domains via GPLVM on hyperbolic manifolds that hyperbolic latent spaces preserve the hierarchical relationships between different robot motions. Similarly, in APo-VAE: Text Generation in Hyperbolic Space, Shuyang Dai and Zhe Gan demonstrate that hyperbolic geometry mitigates "posterior collapse" in text generation by providing a more natural representation for the hierarchical structure of language.
By moving from flat to curved spaces, we transition from models that fight against the data's geometry to models that leverage it.