A neural network cannot answer faster than its deepest chain of layers allows. Each layer waits for the one before it, so single-image latency is bounded by depth no matter how many GPUs are available. "Non-deep" networks get around this by replacing one tall stack with a few short streams that run side by side on separate devices and merge only at the end. Their weakness is cost. To match a 50-layer ResNet with only 12 layers, they need very wide layers and many more parameters. We propose to test whether hyperbolic geometry can pay part of that cost. Hyperbolic geometry is a curved space in which the room available grows exponentially with distance from the center, so tree-like structure fits into few dimensions. The hypothesis is that some of what depth buys in a vision network is hierarchical composition: going from parts to objects to categories. A space that can hold hierarchies without crowding them together might let a shallow network represent that composition without the width.
The concrete proposal is a fully hyperbolic version of ParNet. Its layers are built from recent fast Lorentz-model linear layers, and its parallel streams are fused with a closed-form hyperbolic average rather than Euclidean addition. We ask one question. At matched parameter count and matched batch-1 multi-GPU latency, does curvature flatten the curve of accuracy against depth, and does it do so specifically on the coarse, hierarchical part of the errors? If the answer is yes, curvature and depth are partly interchangeable resources. If it is no, that negative result bounds what hyperbolic inductive bias can do for efficient inference.
Core Claim
We claim that a 12-layer multi-stream network whose streams live on the Lorentz hyperboloid will lose less accuracy per removed layer than an equal-parameter Euclidean ParNet. We also claim the recovered accuracy will show up mainly as fewer "severe" mistakes, meaning confusions across superclasses, rather than as uniform gains. The design keeps the parallel-latency advantage of shallow networks because the added hyperbolic cost is removed at inference time, not merely tolerated.
Three prior results make this testable now rather than speculative. Non-deep Networks supplies the architecture and the gap. ParNet reaches 80.72% ImageNet top-1, 96.12% on CIFAR-10 and 81.35% on CIFAR-100 at depth 12, with ParNet-L running at 4.01 ms on 3 GPUs. It does so with more parameters and FLOPs than deep networks of equal accuracy, and it works only in flat space. Fast and Geometrically Grounded Lorentz Neural Networks supplies the method. It fixes a flaw where earlier Lorentz layers needed exponentially many steps to push features far from the origin. It also shows that the expensive weight transport depends only on parameters and can be cached. With these fixes a hyperbolic ResNet-18 matches Euclidean accuracy on CIFAR-10 (94.66% vs 94.58%) at near-Euclidean inference speed. The embedding literature, from Nickel et al. onward, together with the finding in APo-VAE that distance from the origin tracks generality, supplies the reason to expect hierarchy-specific gains.
How This Differs From the Closest Work
The nearest existing work combining the two directions is HELM. Its Mixture-of-Curvature Experts places several experts with different curvatures inside each block of a 1B-parameter hyperbolic language model, gaining up to 4% on MMLU and ARC-Challenge. It runs about 1.5x slower than Euclidean models and never studies depth, latency scheduling, or multi-device execution. Distributing those experts across GPUs would amount to standard expert parallelism, so we do not pursue it.
Hypformer makes hyperbolic attention linear-time and removes repeated tangent-space mappings. It parallelizes over tokens but keeps a strictly sequential layer stack. FGG-LNN speeds up individual hyperbolic layers but only in deep sequential ResNets.
CQIL runs similar adjacent transformer layers concurrently, cutting LLaMA-33B latency by 48.3% with a drop from 71.7% to 69.3% on downstream tasks. It is a post-hoc surgery on an existing deep Euclidean model and merges branches by flat addition.
Our proposal is different in kind. It changes the geometry of a network designed from the start to be shallow, and it asks whether that geometry substitutes for the depth that was removed.
Research Program
The model, which we call H-ParNet, keeps ParNet's topology: three streams at different resolutions, depth 12, and a final fusion block. Every convolution and linear map in it is an FGG-LNN hyperplane-grounded Lorentz layer, together with that paper's Lorentzian activations. Hypformer's recipe of applying normalization and dropout only to the space-like coordinates handles the remaining operations.
The streams are fused with the Lorentzian centroid, which needs no exponential or logarithm maps. Suppose stream outputs $x_i$ lie on the hyperboloid $\langle x, x\rangle_L = -1/\kappa$, where $\langle\cdot,\cdot\rangle_L$ is the Minkowski inner product and $\kappa>0$ is the curvature magnitude. Let the learned fusion weights be $w_i \ge 0$ and write $s = \sum_i w_i x_i$. The fused point is
$$m = \frac{s}{\sqrt{\kappa}\,\sqrt{|\langle s, s\rangle_L|}},$$
which lies exactly on the manifold because a positive combination of forward time-like vectors is itself time-like. The operation costs one weighted sum and one normalization, so fusion is no more expensive than ParNet's.
Communication between devices also does not grow. The time-like coordinate of each point is determined by its spatial coordinates, so each GPU sends a stream output of exactly the Euclidean size. This matters because interconnect overhead is the known failure mode of intra-layer parallelism, as DISCO documents for edge devices.
After training, we apply FGG-LNN's cache of transported weights. The inference cost of each stream is then a GEMM plus cheap elementwise corrections.
On top of this base model we study one secondary knob, a separate learned curvature per stream. The idea is that the low-resolution stream may encode coarser, more tree-like structure and prefer stronger curvature. HELM's results on curvature diversity make this plausible, but it is an ablation, not the core claim.
The central analysis is a depth sweep. We train Euclidean ParNet and H-ParNet at depths from 6 to 24, with parameters matched at each depth by adjusting width. For each model family we report the accuracy drop $\Delta(d) = A(d_{\max}) - A(d)$, where $A(d)$ is top-1 accuracy at depth $d$. The prediction is that H-ParNet's $\Delta(d)$ is smaller at shallow $d$.
We break every error down using the label hierarchy: the 20 superclasses for CIFAR-100 and WordNet for ImageNet. We then report the fine-level error rate and the mean hierarchical distance of mistakes separately.
We also check whether the geometry is actually doing hierarchical work. To do so we measure whether the hyperbolic norm of final features rises from superclass prototypes to fine classes, following the APo-VAE interpretation of norm as generality.
Baselines
The naive reference point is the Euclidean ParNet reimplemented at the same depth, width budget, data augmentation, epochs, and optimizer-tuning budget as H-ParNet. For each architecture we give both models the same small hyperparameter search over learning rate and weight decay. This is the "do nothing" alternative: keep flat geometry and accept the width. Beating it means higher top-1 accuracy at equal parameters, or equal accuracy with at least 25% fewer parameters, at no worse batch-1 latency on 3 GPUs. As a sanity anchor, our reimplementation should approach ParNet's reported 81.35% on CIFAR-100 and 96.12% on CIFAR-10 before any comparison is trusted.
The strongest existing competitors come in two kinds. For accuracy per parameter, the competitor is a deep hyperbolic network, the FGG-LNN ResNet-18, which reports 94.66% on CIFAR-10. We found no CIFAR-100 accuracy for FGG-LNN reported on Lacuna, so we will rerun it ourselves under our pipeline. For accuracy per unit of latency, the competitor is the published ParNet at its reported operating points. FGG-LNN is the strongest competitor on geometry because it already achieves Euclidean-level accuracy in hyperbolic space. The question is whether collapsing it to depth 12 across parallel streams keeps that accuracy while cutting latency. Beating it means matching its accuracy at lower batch-1 latency. On ImageNet, the bar is the reported ParNet-XL figure of 80.72%, which we will attempt only for the single best configuration found on CIFAR-100 and a ParNet-S-scale ImageNet run.
The key ablation isolates where the curvature matters. It is a Euclidean ParNet body with a hyperbolic classifier head, with parameters matched. A second ablation replaces Lorentzian-centroid fusion with fusion by mapping to the tangent space, averaging, and mapping back. It tests whether the cheap fusion loses accuracy, and it measures the latency the centroid saves.
Why not just…?
Why not just widen the Euclidean ParNet, as the original paper does? That is precisely the trade-off whose cost ParNet itself reports: more parameters and FLOPs, and larger feature maps to move between GPUs. The matched-parameter depth sweep answers the question directly. If Euclidean width closes the gap as efficiently as curvature, the two $\Delta(d)$ curves will overlap and the claim fails. We will report that outcome as a result, not rescue it.
Why not just put a hyperbolic output layer on an ordinary shallow network? This is the most serious threat. Much of the reported benefit of hyperbolic vision models may come from the classifier geometry alone. The head-only ablation settles it. If a Euclidean body with a hyperbolic head recovers most of H-ParNet's gain and most of its reduction in severe mistakes, then curvature is not standing in for depth in the feature hierarchy. In that case we would recommend the cheaper head-only design and narrow our claim accordingly.
Why not keep a deep hyperbolic network and split each layer across GPUs with tensor parallelism? CQIL reports that tensor parallelism gives poor latency gains at batch size 1 because every layer requires communication. The depth-bound latency floor stays in place regardless. We will measure this directly by running FGG-LNN ResNet-18 with within-layer splitting on the same 3-GPU node, comparing batch-1 latency against H-ParNet at equal accuracy.
Why not distill a deep teacher into a shallow Euclidean ParNet? Distillation is a real alternative for shallow students, and it is orthogonal to geometry. We will distill the same deep teacher into both Euclidean ParNet and H-ParNet. If geometry still helps under distillation, the two effects add up. If distillation erases the difference, the geometric claim does not hold in practice.
Decisive Checks
The experiment that decides the proposal is the CIFAR-100 depth sweep at matched parameters, with errors broken down by superclass. The core claim survives only if three things hold together. H-ParNet's accuracy drop at depth 12 or below must be smaller than Euclidean ParNet's. The reduction must be concentrated in cross-superclass errors. And the head-only ablation must recover clearly less of that gain.
CIFAR-10, whose label hierarchy is shallow, serves as a control. The prediction there is a small or zero advantage. A large CIFAR-10 gain would suggest the improvement comes from optimization rather than hierarchy.
We also measure batch-1 latency on a 3-GPU node, comparing the cached H-ParNet against Euclidean ParNet with identical stream placement. The acceptable overhead is at most 10%. Anything above that erodes the reason for building a parallel architecture.
If the claim holds on CIFAR-100, a single ImageNet run and a COCO detection transfer following ParNet's detection protocol will test whether the gain scales to the benchmarks where ParNet's width cost is largest.
References
- Non-deep Networks
- Fast and Geometrically Grounded Lorentz Neural Networks
- Poincaré Embeddings for Learning Hierarchical Representations
- APo-VAE: Text Generation in Hyperbolic Space
- HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts
- Hypformer: Exploring Efficient Hyperbolic Transformer Fully in Hyperbolic Space
- CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers
- DISCO: Distributed Inference with Sparse Communications