Hyperbolic neural networks represent data in a curved space where volume grows exponentially with distance from the center. That makes them a natural fit for tree-like data such as taxonomies, lineages and citation graphs. They are also notoriously hard to train. Gradients explode as representations drift toward the edge of the space, runs end in NaN errors, and 16-bit arithmetic is close to unusable. This proposal argues that much of this fragility is not a property of any single hyperbolic layer. It comes from standard backpropagation, which chains every layer's error signal through every layer above it, so curvature-dependent gradient amplification compounds with depth.
We propose to cut that chain. A fully hyperbolic network is split into a few blocks separated by stop-gradients (operations that block error signals from flowing backward), and each block learns from its own local hyperbolic loss. This is blockwise local learning, previously studied only in flat Euclidean space. The two-sentence version is this. If hyperbolic instability grows with the length of the backward path, then training each block on a short, local path should make deep hyperbolic networks stable in low precision. As a side benefit, compact hyperbolic local losses may ease the "first-block bottleneck" that hurts Euclidean local learning.
Core Claim
Consider standard backpropagation. Let $h_l$ be the output of layer $l$, $J_k$ the Jacobian of layer $k$ (the matrix of partial derivatives of its output with respect to its input), and $L$ the network depth. The error reaching layer $l$ is bounded by
$$\|\nabla_{h_l}\mathcal{L}\| \le \|\nabla_{h_L}\mathcal{L}\| \prod_{k=l+1}^{L} \|J_k\|.$$
In hyperbolic layers, $\|J_k\|$ depends on where the representation sits. In the Poincaré ball, the conformal factor $2/(1-c\|x\|^2)$ blows up near the boundary, where $c$ is the curvature magnitude and $x$ the embedding, as analyzed by Law et al.. In the Lorentz model, the analogous source is the hyperbolic functions (cosh, sinh) that grow with the embedding's radius. Two further results make matters worse. Embeddings migrate outward during training (Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin), and the float precision needed for geometric fidelity grows exponentially with distance from the origin (Yu). Together these imply that the product above grows roughly exponentially in depth times radius. With stop-gradients, the product runs only to the end of layer $l$'s own block, so the worst-case amplification is bounded by block length rather than network depth.
The claim to be tested has two parts. First, a blockwise, fully Lorentz network has lower divergence rates and can train in BF16 or FP16 where the end-to-end version cannot. Second, it does so at a smaller accuracy cost than the roughly one-point gap Euclidean blockwise learning pays on ImageNet.
The first part is the new contribution. The file's search found no paper combining hyperbolic networks with decoupled training. The nearest works each own one half of the problem. Fully intrinsic Lorentz architectures (Intrinsic Lorentz Neural Network, Fully Hyperbolic Convolutional Neural Networks for Computer Vision, Hypformer) fix instability inside layers but always train end to end. Blockwise methods (Blockwise Self-Supervised Learning at Scale, Training Neural Networks with Local Error Signals) cut the backward chain for memory and parallelism, never for numerical stability, and only in Euclidean space.
Research Program
Base architecture. The base model is the hyperbolic ResNet-18 from Fully Hyperbolic Convolutional Neural Networks for Computer Vision, using its public code (kschwethelm/HyperbolicCV). We replace its iterative batch normalization with the closed-form GyroLBN from Intrinsic Lorentz Neural Network. A closed form matters here because iterative Fréchet-mean solvers interact badly with blocks that evolve independently.
Blocks and training schedule. The network is split at its four stages. Following Blockwise Self-Supervised Learning at Scale, all blocks train simultaneously on each forward pass rather than greedily one after another.
Local losses. Each block gets a small Lorentz head. In the supervised setting, the local loss is a Lorentz multinomial logistic regression, plus a geodesic version of the similarity-matching term of Training Neural Networks with Local Error Signals. That paper matches pairwise cosine similarities of activations to label similarities. We instead match pairwise hyperbolic distances $d_{\mathbb{L}}(z_i, z_j)$ between pooled block embeddings $z_i, z_j$ to a two-level target distance from the CIFAR-100 superclass/class tree:
$$\mathcal{L}_{\text{block}} = \mathcal{L}_{\text{MLR}} + \beta \sum_{i,j} \big(d_{\mathbb{L}}(z_i,z_j) - \tau_{ij}\big)^2.$$
Here $\tau_{ij}$ is small for same-class pairs, intermediate for pairs sharing a superclass, and large otherwise, and $\beta$ weights the term. In the self-supervised setting, the local loss is the Lorentz analogue of the hyperbolic InfoNCE of Hyperbolic Contrastive Learning, applied per block.
Radius control. Because local heads could push embeddings to the boundary faster, each block gets a learned radius cap, borrowing the learned-scale RMSNorm idea of HYPER++. We log the cap's value so we can tell whether capacity is being lost, which is the known cost of clipping in Hyperbolic Vision Transformers.
Instrumentation. The analytic side of the program measures the quantity in the bound directly. For every layer we log the per-layer Jacobian norm estimate, the gradient norm reaching each layer, the embedding radius distribution, and the step at which any non-finite value first appears. We do this across FP32, BF16 and FP16 and across depths (ResNet-18, -34, -50). The central prediction is falsifiable. Under end-to-end training, the gradient norm reaching early layers should grow with depth and with mean radius, and divergence should concentrate in low precision. Under blockwise training, both effects should flatten.
Baselines
Naive reference points. There are two. The first is the Euclidean ResNet-18 trained end to end, which reaches 77.72% on CIFAR-100 as reported in both hyperbolic vision papers. The second is the same hyperbolic network trained end to end with no extra stabilization. Its purpose is to show the size of the instability problem in low precision rather than to compete on accuracy.
Strongest hyperbolic competitor. This is the end-to-end fully hyperbolic CNN. Fully Hyperbolic Convolutional Neural Networks for Computer Vision reports 78.76% CIFAR-100 accuracy for its hybrid HECNN, and Intrinsic Lorentz Neural Network reports 78.41% for ILNN. To be fair, we give this baseline every stabilizer our method gets, and more. It receives the same GyroLBN, the same learned radius cap, global gradient-norm clipping tuned over the same grid, and the same seeds, epochs and learning-rate search budget. This is the baseline most likely to erase our advantage.
Beating it means one of two outcomes. Either blockwise training matches its accuracy within one point in FP32 while showing a clearly lower divergence rate in BF16/FP16 across at least five seeds, or it trains to competitive accuracy in a precision where the end-to-end model diverges. No reported result was found on Lacuna for hyperbolic CNNs trained in BF16 or FP16, so we establish those numbers ourselves.
Strongest decoupled-training competitor. This is Euclidean blockwise learning. For self-supervision, Blockwise Self-Supervised Learning at Scale reports a 70.48% ImageNet-1K linear probe with a four-block ResNet-50, versus 71.57% end to end. It relies on channel expansion from 256 to 2048 dimensions in its local heads. For supervised local learning on CIFAR-10, Training Neural Networks with Local Error Signals reports 3.6% error. We rerun both on our block split with FLOP-matched heads: the Euclidean head's expansion width is set so its compute equals our Lorentz head's.
Ablations. Two ablations isolate the key ingredients. The first keeps the hyperbolic network and the hyperbolic local losses but removes the stop-gradients. The loss is identical; only the backward chain length changes. If stability comes from chain truncation, this variant should behave like end-to-end training. The second makes every block Euclidean except the last, to test whether hyperbolic geometry is needed in early blocks at all.
Why not just…?
Why not just stabilize end-to-end hyperbolic training with clipping and normalization? This is the real threat. HYPER++ shows that RMSNorm with learned scaling plus the Lorentz model stops gradient blowup in reinforcement learning, and ILNN's intrinsic layers fixed the collapse of earlier hyperbolic models on Covid variant classification (64.76% versus 14.81%). If these fixes remove low-precision divergence in deep end-to-end networks too, our stability claim fails. The fully stabilized end-to-end baseline in BF16/FP16 at depths 18 to 50 settles this directly. If the gap vanishes, we report that blockwise training adds nothing for stability, and the proposal reduces to its memory and parallelism benefits.
Why not just use Euclidean blockwise learning with wider heads? Width does work: the 256-to-2048 expansion is part of why the Euclidean blockwise method gets within about a point of backpropagation. Hyperbolic embeddings keep hierarchy in 8 to 32 dimensions where Euclidean ones collapse (Fully Hyperbolic Convolutional Neural Networks for Computer Vision, Hyperbolic Vision Transformers). That suggests compact hyperbolic heads could match wide Euclidean ones at lower cost, but Lorentz arithmetic is slower per operation, a cost ILNN concedes. The FLOP-matched comparison, reported against both accuracy and wall-clock time, decides it. If Euclidean heads win at equal compute, the accuracy argument for hyperbolic local losses is dead, though the stability question about hyperbolic backbones remains separate.
Why not just go hybrid, with Euclidean early blocks and a hyperbolic final block? HCNN found hybrid encoders performed best and that early layers learn flat, low-level features. That is a sensible prior against hyperbolic early blocks. The Euclidean-except-last ablation tests it. If it matches the fully hyperbolic blockwise model, we adopt it as the recommended design. Blockwise training would then still matter only if the final hyperbolic block is where divergence originates.
Why not just formulate hyperbolic Direct Feedback Alignment? Direct Feedback Alignment sends the output error to every layer through fixed random matrices, so no layer waits for the one above it. It is the purest form of cutting the chain. However, it already trails backpropagation on convolutional vision in flat space: 26.9% versus 22.5% CIFAR-10 error in Direct Feedback Alignment Provides Learning in Deep Neural Networks. It also needed careful optimizer tuning even where it scaled (Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures). On a curved space, a random linear projection of the error does not even land in the right tangent space. Stacking two unsolved problems would make any failure uninterpretable, so we leave hyperbolic feedback alignment as follow-up work once the stability effect of truncation is established with the simpler blockwise method.
Proposed Experiments
Supervised CIFAR-100 (main testbed). We run CIFAR-100 with hyperbolic ResNet-18/34/50 across three precisions, three training regimes (end to end, stabilized end to end, blockwise) and five seeds. The metrics are top-1 accuracy, divergence rate, the per-layer gradient-norm profile and radius distributions. We also measure superclass consistency of errors, meaning the fraction of misclassifications that stay within the correct superclass, to check whether local geodesic similarity matching preserves the two-level hierarchy.
Self-supervised learning. On CIFAR-100, blockwise Lorentz contrastive learning is compared against the 55.89% linear-evaluation accuracy of Hyperbolic Contrastive Learning. If CIFAR-scale results are positive, we scale to a four-block ResNet-50 on ImageNet-1K against the 70.48% Euclidean blockwise and 71.57% end-to-end figures.
Instability stress test. We use the genomic Covid variant task from ILNN, where a prior hyperbolic model collapsed, to test whether blockwise training recovers accuracy without ILNN's full set of intrinsic layers.
Residual connections. We check whether the geodesic similarity loss survives identity skip connections. Training Neural Networks with Local Error Signals reports that its Euclidean version fails on ResNets unless the skip connections are removed.
Decisive check. The single most informative result is the stop-gradient ablation in BF16 at depth 50. If removing stop-gradients while keeping everything else fixed brings divergence back and restores the depth-dependent growth of early-layer gradient norms, then the backward chain, not the local loss, is the stabilizing factor. That is the core claim of this note.
References
- Lorentzian Distance Learning for Hyperbolic Representations
- Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin
- Numerically Accurate Hyperbolic Embeddings Using Tiling-Based Models
- Intrinsic Lorentz Neural Network
- Fully Hyperbolic Convolutional Neural Networks for Computer Vision
- Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Space
- Blockwise Self-Supervised Learning at Scale
- Training Neural Networks with Local Error Signals
- Hyperbolic Contrastive Learning
- Understanding and Improving Hyperbolic Deep Reinforcement Learning
- Hyperbolic Vision Transformers: Combining Improvements in Metric Learning
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures