Hyperbolic geometry has emerged as a powerful tool for representation learning, particularly for hierarchical or tree-structured data. Because the volume of hyperbolic space grows exponentially with its radius, it naturally accommodates the exponential growth of nodes in a tree. This allows models to embed complex hierarchies with exceptionally low dimensional distortion.
However, a fundamental engineering and theoretical challenge lurks beneath this geometric harmony: the precision wall. As hierarchical data grows deeper and larger, the numerical representation of these embeddings breaks down under standard computer arithmetic, causing distinct data points to collapse into identical coordinates.
The Geometry of Crowding
To understand why precision becomes a bottleneck, we must look at how trees are mapped into hyperbolic models, such as the Poincaré ball. The Poincaré ball represents hyperbolic space within a bounded unit sphere. The origin represents the root of the hierarchy, and as you move outward toward the boundary (where the radius $r \to 1$), the metric distance stretches exponentially.
Euclidean Coordinate Space (Poincaré Disk)
.---.
/ o \ <- Root at origin
| / \ |
| o o | <- Intermediate levels
|/ \ / \|
'-------' <- Leaves crammed exponentially close to the boundary (r -> 1)
To preserve the true distances between nodes in a deep tree, the edges must maintain a relatively consistent hyperbolic length. Because the metric space expands exponentially near the boundary, the corresponding Euclidean coordinates of these nodes must be placed infinitesimally close to the boundary and to each other.
As Sarkar demonstrated, achieving near-zero distortion in hyperbolic tree embeddings inherently requires long edge lengths, which forces coordinates to approach the boundary. In this regime, the precision required to distinguish between distinct sibling nodes or parent-child pairs increases exponentially with the depth of the tree.
Sala et al. formalized this relationship, proving a fundamental tradeoff: the numerical precision (in bits) required to represent a hierarchy scales linearly with the depth of the tree. When embedding deep hierarchies, the limiting factor is no longer the dimensionality of the space, but the bit-width of the coordinate system.
Why Standard Floating-Point Fails
In modern deep learning pipelines, standard 32-bit floating-point (FP32) representations are the default. An FP32 number allocates 23 bits to its significand (mantissa), offering roughly 7 decimal digits of precision. 64-bit floating-point (FP64) offers about 15–17 decimal digits.
When optimizing or constructing embeddings for deep hierarchies, these limits are quickly breached:
- Node Collapse: As nodes move closer to the boundary to represent deeper levels, the available floating-point bits cannot resolve the tiny spatial differences between separate leaves. As noted by Sonthalia et al. and Dong et al., this lack of resolution causes distinct nodes to collapse into identical numerical coordinates.
- The NaN Problem: Optimization algorithms rely on computing gradients of hyperbolic distances. As points crowd near the boundary, calculating these distances involves terms like $1 - \|x\|^2$. When $\|x\|$ is extremely close to 1, floating-point underflow or catastrophic cancellation occurs, driving denominators to zero and triggering gradient overflow or "NaN" errors (Yu et al.).
Tan et al. evaluated this gap empirically, showing that standard FP32 precision prevents Hyperbolic Neural Networks from reaching the theoretical representation limits that continuous hyperbolic geometry promises.
Strategies for Breaking the Precision Wall
To scale hyperbolic embeddings to massive, deep hierarchies without suffering from representation collapse, researchers have developed three primary paradigms: coordinate remapping, discrete tiling, and hardware-level extended precision.
1. Coordinate Remapping (Polar and Infinite Boundaries)
One way to avoid the boundary bottleneck of the Poincaré ball is to change how the space is parameterized. Wang et al. introduced the Hyperbolic extended Poincaré Ball Embedding (HBE), which utilizes a polar coordinate system and extends the boundary to infinity. By decoupling the radial depth from a bounded interval, the model mitigates the extreme entity crowding and floating-point precision errors typical of standard models.
2. Tiling-Based Models
Instead of relying on a single global floating-point coordinate system, one can discretize the space. Yu et al. proposed a tiling-based model inspired by hyperbolic tessellations.
Tiling Approach:
+-------------+-------------+
| Tile 0 | Tile 1 |
| [Integer] | [Integer] |
| o (local) | |
+-------------+-------------+
| Tile 2 | Tile 3 |
| | o (local) |
+-------------+-------------+
In this framework, a point's position is determined by a combination of an integer-based "tile" identifier and a local floating-point coordinate within that tile. Because the floating-point component only needs to represent positions relative to a local origin, the numerical error remains provably bounded everywhere in the space, completely eliminating the gradient overflow problem.
3. Floating-Point Expansions on Hardware
While combinatorial methods can construct perfect tree embeddings offline, they often fail to integrate with modern deep learning hardware like GPUs, which are optimized for standard floating-point formats.
To bridge this gap, van Spengler et al. (ICML 2025) and van Spengler et al. (ICLR 2025) introduced HypFPE. This method implements floating-point expansions directly into the hyperbolic distance and inversion routines. By representing an ultra-high-precision number as the sum of multiple standard floating-point numbers, they enable arbitrary-precision hyperbolic math natively on GPUs. This prevents leaf node collapse while maintaining compatibility with standard deep learning optimization pipelines.
Key Foundational Work
Representation Tradeoffs for Hyperbolic Embeddings (2018) — contributed 4 concept elements to this direction.
Low Distortion Delaunay Embedding of Trees in Hyperbolic Plane (2011) — contributed 1 concept elements to this direction.