Most deep learning models operate under an implicit assumption: the latent spaces where data is embedded are flat and Euclidean ($\mathbb{R}^n$). We default to Euclidean space because its mathematical properties—like vector addition and standard inner products—are computationally convenient and intuitive.
However, as highlighted by Weber et al., this foundational assumption creates a severe geometric inductive bias mismatch when dealing with complex, real-world datasets. Relational networks, biological taxonomies, molecular structures, and cyclical patterns possess intrinsic non-Euclidean geometries. Forcing these structures into a flat space distorts distances, collapses hierarchical relationships, and reduces downstream model performance.
To overcome this limitation, a powerful paradigm has emerged: replacing flat Euclidean embeddings with mixed-curvature product manifolds that adaptively match the underlying geometry of the data.
The Intuition of Volume Growth and Distortion
To understand why flat spaces fail, consider how different data structures expand.
In a hierarchical tree structure (such as a word taxonomy or a citation network), the number of nodes grows exponentially with the depth of the tree. If a node has a branching factor of $b$, the number of leaves at depth $d$ is $b^d$.
Now, consider the capacity of a space to hold these nodes. In a standard $n$-dimensional Euclidean space, the volume of a ball of radius $r$ scales polynomially:
$$\text{Volume}_{\text{Euclidean}}(r) \propto r^n$$
When you try to force an exponentially growing tree into a polynomially growing space, you run out of room. To maintain the true graph distances between connected nodes, the representation must squeeze sibling nodes closer and closer together as you move down the hierarchy. This crushing effect causes massive distortion and numerical instability.
Constant-Curvature Alternatives
To resolve this geometric mismatch, we can turn to non-Euclidean Riemannian manifolds with constant sectional curvature $\kappa$:
- Hyperbolic Space ($\kappa < 0$): In hyperbolic space, the volume of a ball grows exponentially with its radius ($\text{Volume}(r) \propto e^{r}$). This perfectly mirrors the exponential branching of trees. Hyperbolic space can be thought of as a continuous tree, allowing hierarchical data to be embedded with near-zero distortion.
- Spherical Space ($\kappa > 0$): Spherical space is bounded and wraps around itself. This geometry is naturally suited for cyclical, periodic, or directional data patterns, where moving continuously in one direction eventually brings you back to the starting point.
As shown by Alippi et al. and Fonio et al., leveraging these constant-curvature spaces within generative and prototype-based models significantly improves representation efficiency for purely hierarchical or purely cyclical datasets.
Euclidean (\kappa = 0) Hyperbolic (\kappa < 0) Spherical (\kappa > 0)
Flat Grid Exponential Growth Bounded Loops
[Polynomially Fits] [Exponentially Fits] [Periodically Fits]
Continuous Data Hierarchies Cycles
The Reality of Real-World Data: Product Manifolds
While constant-curvature spaces are an improvement, real-world datasets are rarely purely hierarchical or purely cyclical. For instance, a molecular graph might feature an overall tree-like hierarchical backbone but contain localized carbon rings (cycles). Forcing such a dataset entirely into hyperbolic space would distort the cycles, while forcing it into spherical space would distort the hierarchy.
The solution is to construct a product manifold—a composite latent space formed by the Cartesian product of multiple lower-dimensional spaces, each with its own distinct curvature:
$$\mathcal{M} = \prod_{i=1}^{k} \mathcal{M}_i^{\kappa_i}$$
By factorizing the latent space into a combination of Euclidean ($\mathbb{R}$), hyperbolic ($\mathbb{H}$), and spherical ($\mathbb{S}$) components, a model can simultaneously isolate and preserve different structural facets of the data.
Skopek et al. pioneered this approach with the Mixed-curvature Variational Autoencoder (M-VAE). Instead of mapping an image or a graph to a flat vector, the encoder maps data to coordinates across a product of curved manifolds. The model learns not only the coordinates but also the optimal curvature for each component space.
This multi-space perspective has proven transformative across machine learning paradigms:
- Graph Inference: Sáez de Ocáriz Borde et al. demonstrated that inferring latent graphs within product manifolds captures complex, hybrid node relationships far better than Euclidean graph learning.
- Classical Classifiers: Moving beyond deep networks, Chlenski et al. extended decision trees and random forests to operate directly on product manifolds by reimagining data splits as angles within curved 2D subspaces.
Automating Latent Geometry Search
If the optimal latent space is a combination of different geometries, how do we determine the exact recipe? How many dimensions should be hyperbolic, how many spherical, and what should their curvatures be?
One strategy is dynamic routing. Zhang et al. introduced a data-informed space selection framework that uses a sparse gating mechanism. Instead of fixing the geometry globally, the model dynamically routes individual data points to the geometric space (Euclidean, hyperbolic, or spherical) that best matches their local structural characteristics.
Another strategy treats the latent geometry as a global discrete-continuous optimization problem. Sáez de Ocáriz Borde et al. developed Neural Latent Geometry Search (NLGS). This framework uses the Gromov-Hausdorff distance—a mathematical metric that measures how far two metric spaces are from being isometric—to evaluate how well a candidate product manifold matches the intrinsic distances of the input data. NLGS then navigates the combinatorial space of possible manifold configurations using Bayesian Optimization to automatically discover the ideal latent shape for a given task.
Similarly, in few-shot learning scenarios where data is scarce, Gao et al. showed that models can dynamically generate task-aware curved embedding spaces, adapting their geometry on the fly to the specific taxonomy of the target classes.
Redesigning the Deep Learning Toolkit
Operating entirely within non-Euclidean spaces requires rewriting the fundamental building blocks of deep learning. Standard operations like $Wx + b$ assume vector space properties that do not hold on curved surfaces; you cannot simply add two points on a sphere or a hyperboloid without flying off the manifold.
To resolve this, researchers utilize gyrovector spaces—an algebraic framework that generalizes vector addition and scalar multiplication to curved domains. As shown by Nguyen et al., gyrovector calculus allows for the principled construction of neural network layers (like Multiclass Logistic Regression) directly on complex matrix manifolds, such as Symmetric Positive Definite (SPD) and Grassmann manifolds. This mathematical foundation was further extended by Nguyen et al. to build deep architectures on Siegel spaces, which generalize constant-curvature spaces to handle multi-channel signal and network topologies.
For sequence and graph architectures, where structural mismatches are particularly damaging (Ju et al.; Rozemberczki et al.), operations must be generalized to product spaces without sacrificing computational efficiency. Cho et al. introduced the Fully Product-Stereographic Transformer (FPS-T), which redefines the attention mechanism using stereographic projections. This allows the Transformer to compute kernelized attention entirely within mixed-curvature spaces, preserving linear time complexity while respecting the graph's underlying geometry.
Downstream Advantages: Stability and Efficiency
Matching the model's geometry to the data's natural structure provides strong inductive biases that yield practical benefits beyond raw accuracy:
- Mitigating Catastrophic Forgetting: In continual learning, models often overwrite past knowledge when learning new tasks. Sun et al. and Roy et al. demonstrated that adaptive Riemannian and mixed-curvature spaces anchor representations structurally. Because the geometry natively accommodates evolving hierarchies, new tasks can be integrated into the latent space without collapsing or distorting previously learned structures.
- Robustness under Label Scarcity: When labels are extremely limited, flat spaces easily overfit or fail to separate classes. Dong et al. showed that projecting graph data into an ensemble of Euclidean, hyperbolic, and spherical spaces allows for highly accurate node anomaly detection with minimal supervision, because the multi-space distances inherently expose structural anomalies.
- Simplified Generative Architectures: Traditional VAEs rely on complex encoder networks to map data to latent distributions. By respecting the intrinsic manifold geometry, the Riemannian Generative Decoder (RGD) proposed by Bjerregaard et al. eliminates the encoder entirely. Treating latent variables as direct parameters on curved manifolds combined with geometry-aware noise achieves superior scalability and interpretability.
By breaking free from the flat-space paradigm and embracing the geometry of product manifolds, modern neural architectures achieve a more compact, expressive, and mathematically rigorous representation of the real world.
Key Foundational Work
Adversarial Autoencoders with Constant-Curvature Latent Manifolds (2018) — contributed 1 concept elements to this direction.
Mixed-curvature Variational Autoencoders (2020) — contributed 1 concept elements to this direction.