When embedding complex relational data—such as hierarchical trees, social networks, or knowledge graphs—standard flat Euclidean space often feels claustrophobic. Because the number of nodes in a tree grows exponentially with depth, embedding a tree into a low-dimensional Euclidean space forces parent and child nodes to crowd together, causing massive geometric distortion.
To solve this, researchers frequently turn to non-Euclidean geometries. Hyperbolic space, for instance, expands exponentially as you move away from the origin, providing a natural mathematical home for branching hierarchical structures.
Yet, a recurring and counterintuitive phenomenon emerges when scaling up these models: as the dimensionality of the latent space increases, the hard-won performance advantages of specialized non-Euclidean geometries steadily diminish, eventually converging to match simple Euclidean baselines.
Understanding why this high-dimensional convergence happens requires looking at two distinct forces: the mathematical quirks of high-dimensional manifolds and the capacity saturation of flat spaces.
The Phenomenon of Diminishing Geometric Returns
The performance gap between specialized geometries and flat Euclidean space is most pronounced under strict low-dimensional constraints. As noted by Atsushi Suzuki et al., while hyperbolic embeddings excel at capturing hierarchical rankings in low dimensions, their behavior changes significantly in high-dimensional regimes.
Across a wide variety of domains, this pattern repeats:
- Deep Neural Networks: In a comprehensive survey, Wei Peng et al. highlighted that the relative performance gains of hyperbolic representations over Euclidean ones consistently shrink as embedding dimensionality grows.
- Knowledge Graphs: Models designed to capture complex multi-relational hierarchies, such as those by Ryohei Shimizu et al. and Huiru Xiao et al., show that specialized non-Euclidean manifolds offer diminishing returns when the embedding space is granted more dimensions.
- Generative and Manifold ODEs: Even when modeling continuous trajectories on curved surfaces, Aaron Lou et al. observed that the effectiveness of non-Euclidean inductive biases becomes marginal in high-dimensional latent spaces.
Mechanism 1: Volume Concentration and Boundary Isometry
Why does the unique geometry of a curved space vanish in high dimensions? The primary culprit is a mathematical phenomenon known as volume concentration.
In any high-dimensional ball, the vast majority of the space's volume does not live near the center; it is concentrated in a microscopically thin shell right at the outer boundary. In hyperbolic spaces (like the Poincaré ball), this effect is amplified dramatically because the metric tensor itself expands exponentially toward the edge.
As a result, during the training of a high-dimensional neural network, optimization forces the learned embeddings to migrate outward toward the surface of the space. Gabriel Moreira et al. demonstrated that in high-dimensional hyperbolic models, embeddings inevitably end up trapped at this boundary.
This migration has a devastating geometric consequence: boundary isometry. At the extreme edge of a hyperbolic ball, the hyperbolic geodesic distance becomes functionally equivalent to standard Euclidean angular distance on a hypersphere. By pushing embeddings to the boundary, the high-capacity neural network backbone inadvertently neutralizes the hierarchical representation properties of the hyperbolic manifold, reducing it to a simpler spherical Euclidean model.
Mechanism 2: Euclidean Capacity Saturation
The second reason for this convergence is that high-dimensional Euclidean space simply becomes "roomy" enough to handle hierarchies on its own.
A low-dimensional Euclidean space cannot fit an exponential tree because it lacks the volume. However, as you add dimensions, the space gains an enormous number of mutually orthogonal directions. This explosion of degrees of freedom allows a flat space to approximate or implicitly accommodate hierarchical structures without needing an explicit geometric prior.
This capacity saturation manifests in several ways during optimization:
- Curvature Flattening: When training flexible models like ATTH, Ines Chami et al. found that the learned curvature of the hyperbolic space often approaches zero in high-dimensional settings, effectively flattening the manifold back into Euclidean space.
- Angular Dominance: Peng Wang et al. observed that as dimensionality increases, the hyperbolic curvature matters less, and the angular components of the embedding become dominant.
- Coordinate Convergence: This is mirrored by Ran Iwamoto et al., who showed that while specialized coordinate systems (like polar coordinates) provide a massive advantage for hierarchies in low dimensions, the gap narrows to near-zero in high dimensions because standard Euclidean embeddings naturally find ways to organize hierarchical data when given enough space.
In fact, forcing a model to use a single high-dimensional hyperbolic space can sometimes backfire. Jun Takeuchi et al. noted that scaling up the dimensions of a single hyperbolic space can introduce severe geometric distortions, and Qiuyu Liang et al. documented cases where manifold-based rotations underperform on precision metrics in high dimensions.
A Universal Statistical Pattern
This convergence toward geometric simplicity is not unique to embedding spaces; it is a fundamental property of high-dimensional statistics that appears across multiple domains:
| Domain | Low-Dimensional Behavior | High-Dimensional Convergence |
|---|---|---|
| Locality-Sensitive Hashing (Jianmin Li et al.) | Enforcing strict orthogonality between random projection vectors significantly reduces variance. | Diminishing returns; random vectors in high dimensions are already approximately orthogonal with high probability. |
| SDE Discretization (Masaaki Fukasawa) | Specialized sphere-hitting schemes drastically reduce simulation errors compared to standard Gaussian increments. | The performance advantage scales inversely with dimensionality; the error reduction factor approaches unity. |
| Transformer Attention (Shaked Brody et al.) | Normalization layers (like LayerNorm) alter token reachability and expressivity by constraining geometric layouts. | The statistical probability of a vector falling within the convex hull of others drops naturally, altering the necessity of these constraints. |
Engineering Takeaways
The reality of high-dimensional convergence shifts how we should approach geometric deep learning.
If your application is strictly resource-constrained—such as mobile deployment, edge computing, or compact lookups—specialized non-Euclidean frameworks like those proposed by Paul Berg et al. and Hidetoshi Shimodaira et al. offer massive efficiency gains, compressing complex graphs into tiny latent spaces.
However, if you are already utilizing a high-capacity neural network backbone with large embedding sizes, standard inner products and Euclidean spaces will frequently match the performance of complex geometric configurations (Kirill Sergeevich Shevkunov et al.). In the high-dimensional limit, the intrinsic capacity of the space triumphs over the structural bias of the manifold.
Key Foundational Work
Hyperbolic Ordinal Embedding (2019) — contributed 1 concept elements to this direction.
Efficient discretisation of stochastic differential equations (2019) — contributed 1 concept elements to this direction.
Super-Bit Locality-Sensitive Hashing (2012) — contributed 1 concept elements to this direction.