Standard machine learning models operate under a "flat earth" assumption, treating latent spaces and data features as vectors in Euclidean space $\mathbb{R}^n$. However, many real-world domains possess intrinsic geometric structures that defy flat representations. For instance, hierarchical structures like taxonomies or language trees embed naturally into hyperbolic spaces (Emile Mathieu et al., Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders). Physical rotations inhabit the Lie group $SO(3)$ (Taeyoung Lee et al., Global Symplectic Uncertainty Propagation on SO(3)), and covariance matrices form a manifold of Symmetric Positive Definite (SPD) matrices (Thibault de Surrel et al., Wrapped Gaussian on the manifold of Symmetric Positive Definite Matrices).
When we force these geometries into Euclidean models, we encounter "manifold mismatch" (Luis A. Pérez Rey et al., Diffusion Variational Autoencoders). Standard Gaussian distributions cannot be easily defined on curved surfaces because their density functions do not naturally integrate to one, and they fail to capture the topological constraints of the space.
The Core Mechanism: Tangent Space Wrapping
How do we construct a mathematically rigorous "Gaussian" on a curved manifold? The key intuition relies on a fundamental property of Riemannian manifolds: at any given point, the space is locally flat. This local linear approximation is called the tangent space.
Instead of defining a distribution directly on the curved surface, we define a standard multivariate Gaussian distribution in this flat, Euclidean tangent space. We can then project this distribution onto the manifold using the exponential map ($\text{Exp}_\mu$), which maps vectors from the tangent space to coordinates on the manifold surface while preserving distances along geodesics.
This process is known as wrapping a distribution. As Yoshihiro Nagano et al. demonstrate in A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning, sampling from a wrapped normal distribution involves three elegant steps:
- Sample a vector $v$ from a standard Euclidean Gaussian $\mathcal{N}(0, \Sigma)$ in the tangent space at the origin.
- Transport this vector via parallel transport to the tangent space of the target mean $\mu$.
- Project the vector onto the manifold surface using the exponential map $\text{Exp}_\mu(v)$.
Tangent Space (Flat Euclidean) Curved Manifold Surface
[ v ~ N(0, Σ) ]
|
v (Parallel Transport)
[ v_μ at Mean μ ]
|
+=======( Exponential Map )=======> [ Wrapped Sample on Manifold ]
Why It Works: Capturing Directional Noise and Anisotropy
A major advantage of the wrapping approach over isotropic manifold distributions (such as the von Mises-Fisher distribution) is the ability to model anisotropic data spreads. Isotropic distributions assume that variance is identical in all directions, resulting in perfectly circular probability contours.
By defining the initial distribution in the tangent space, we can utilize a full covariance matrix $\Sigma$. As Julian Straub shows in A Dirichlet Process Mixture Model for Spherical Data, projecting data into local tangent spaces allows models to capture elongated, elliptical cluster shapes on curved surfaces. This capability is crucial for domains like brain-computer interfaces or medical imaging, where directional noise varies widely across different dimensions of SPD matrices (Florian Yger et al., A probabilistic view on Riemannian machine learning models for SPD matrices).
Enabling Gradient-Based Learning and Variational Inference
For deep generative models like Variational Autoencoders (VAEs), a distribution must not only be expressible but also differentiable to allow for backpropagation via the reparameterization trick.
By utilizing the exponential map, wrapped distributions inherit the differentiability of the underlying manifold operations. Ondrej Skopek et al. leverage this in Mixed-curvature Variational Autoencoders to construct latent spaces that combine spherical, hyperbolic, and flat geometries. Similarly, Shuyang Dai et al. use wrapped normal distributions in APo-VAE: Text Generation in Hyperbolic Space to mitigate posterior collapse when modeling the hierarchical nature of text.
The mapping ensures that standard sampling techniques remain compatible with gradient-based optimization across complex topologies, a mechanism also applied to graph embeddings (Ali Lotfi-Rezaabad et al., Hyperbolic graph embedding with enhanced semi-implicit variational inference) and phylogenetic tree reconstruction (Alex Chen et al., Variational Combinatorial Sequential Monte Carlo for Bayesian Phylogenetics in Hyperbolic Space).
Overcoming the Infinite Summation Problem
While wrapping is highly intuitive, computing the exact probability density function (PDF) introduces a significant mathematical challenge on compact or cyclic manifolds (like spheres or rotation groups). Because the exponential map can wrap around the manifold multiple times, multiple distinct points in the tangent space can project onto the exact same point on the manifold. To compute the true density, one must sum the probabilities over an infinite number of tangent space pre-images.
To bypass this computational bottleneck, researchers have developed two primary strategies:
- Compact Support: Sergey Troshin et al. propose using wrapped $\beta$-Gaussians in Wrapped $\beta$-Gaussians with compact support for exact probabilistic modeling on manifolds. Because these distributions have bounded support in the tangent space, the probability mass does not overlap itself when projected, completely eliminating the need for infinite summations.
- Equivolumetric Discretization: When closed-form normalization constants are unavailable, the factor can be approximated by summing over a dense, evenly distributed grid of points directly on the manifold surface, as demonstrated by Yingda Yin et al. for rotation estimation in A Laplace-inspired Distribution on SO(3) for Probabilistic Rotation Estimation.
Alternative Perspectives: Flows and Potentials
Beyond explicitly wrapping a single Gaussian, the principle of mapping flat distributions to curved geometries extends to more complex generative frameworks. For example, Samuel Cohen et al. introduce Riemannian Convex Potential Maps, which transform probability distributions by applying the manifold's exponential map to the gradient of a learned potential function.
Similarly, Heli Ben-Hamu et al. use manifold-specific kernels like the von Mises-Fisher distribution to define target probability paths for continuous normalizing flows (Matching Normalizing Flows and Probability Paths on Manifolds), bypassing the need to solve expensive differential equations globally.
Key Foundational Work
Robust point-mass filters on manifolds (2009) — contributed 1 concept elements to this direction.
Global Symplectic Uncertainty Propagation on SO(3) (2008) — contributed 1 concept elements to this direction.