Curvature at Compile Time: Hyperbolic Joint Layouts for Edge Skeleton Recognition

A human skeleton is a small tree: joints are nodes, bones are edges. Two families of fast action-recognition models each exploit this tree differently. One flattens the skeleton into an image, with time on one axis and joints on the other, so that a cheap 2D convolutional network can read it. These "skeleton images" work only if physically connected joints sit next to each other in the image, which is why researchers order joints by walking the tree. The other family embeds the skeleton in hyperbolic space, a curved geometry whose volume grows exponentially with radius, much as the number of nodes in a tree grows with depth. That geometry represents hierarchies with very little distortion. It is also expensive: every layer runs exponential maps, logarithmic maps, and curved-space addition, and none of these map well onto GPU matrix-multiply hardware.

This note proposes paying for curvature once, offline, instead of at every forward pass. We embed the kinematic tree in the hyperbolic plane with a non-iterative construction. From that embedding we derive two things: a joint ordering with no duplicated joints, and a fixed per-joint coordinate channel. Both feed a purely Euclidean network that runs on standard fused kernels and quantizes easily. The scientific question is sharp and currently unanswered. Is the accuracy that hyperbolic skeleton models report a product of computing in curved space? Or does it come from supplying hierarchical structure at all, which a layout chosen ahead of time could provide at zero runtime cost?

Core Claim

For skeleton action recognition under an edge latency budget, we hypothesize that most of the gain from runtime hyperbolic computation can be recovered by a Euclidean model that receives the hierarchy through its input layout. Two input-side mechanisms carry that hierarchy. The first is the order of joints along the convolution axis. The second is a constant channel holding each joint's offline hyperbolic coordinates (radius for depth in the tree, angle for branch). Once the network is compiled for a device, it should be measurably faster and more robust to low-bit quantization than its runtime-hyperbolic counterpart.

The claim is falsifiable in both directions. If runtime curvature still wins clearly at matched parameters, that is evidence that curved computation, not structural information, is doing the work. If simple Euclidean structural encodings match our hyperbolic coordinates, the right conclusion is "tree structure, not curvature," and we will report it that way.

Why this is not already done

The closest work is HyLiFormer. It projects skeleton sequences into the Poincaré ball (a disk model of hyperbolic space) and computes linear attention there, reaching 91.7% on NTU60 cross-subject with 1.9M parameters and 3.5G FLOPs. It treats joints as an unordered token set, and it names its own back-and-forth Euclidean–hyperbolic mappings as a source of overhead. Our proposal removes curvature from the runtime entirely and makes joint order a first-class variable. HyLiFormer does neither.

HYSP uses the Poincaré radius as a self-paced certainty signal on a graph network. It motivates our radius channel, since radius tracks how coarse or subtle a movement is. However, it never touches input layout or inference cost.

On the ordering side, Tree Structure Skeleton Images, TSRJI and SkeleMotion use depth-first traversals with backtracking. These duplicate hub joints and stay entirely Euclidean. They supply the method we build on. They also supply our motivating evidence: tree order beat random order by up to 5.5 points in TSRJI. Li et al. gives the general argument that convolutions waste capacity on arbitrary layouts.

The constructive tree embedders supply our offline tool. TREEREP builds a tree from a metric and places it in hyperbolic space in linear time. HS-DTE reports worst-case distortion of 1.07 on 3-ary trees. Neither has been applied to kinematic skeletons.

We should be honest about one point. On the fixed 25-joint anatomical tree, sorting joints by angle in a constructive embedding gives a pre-order traversal without repeats. The ordering by itself is not where curvature earns its keep. The contribution lies elsewhere, in three places. The first is the coordinate channel. The second is the data-derived tree, built by TREEREP from joint co-motion distances, which can differ from anatomy. The third and main one is the controlled answer to the runtime question.

Research Program

We first build the layouts. We run a Sarkar-style construction on the anatomical tree, using HS-DTE's angle-maximizing child placement, to give each joint a point in the hyperbolic disk. The tree has depth under ten, so the boundary-precision problem that HS-DTE's extended floating point addresses does not arise; float32 suffices.

Joints are sorted by angle to give a single row per joint. This removes the duplicate rows that depth-first skeleton images carry. Each joint's radius and angle become two constant input channels, broadcast over time.

As a second variant, we compute pairwise distances between joints from training-set velocity correlations and fit a tree to them with TREEREP. Its extra internal "Steiner" nodes act as learned body parts, which are used for both ordering and coordinates. This tests whether a functional hierarchy, such as which joints actually move together during "writing," helps more than anatomy.

Next we build matched backbones in two regimes. In the low-cost regime, we use a TSRJI-style shallow convolutional network, where order matters by construction. In the strong regime, we take HyLiFormer's architecture and replace its hyperbolic projection and attention with Euclidean linear attention plus a depthwise convolution along the joint axis. Parameters are held within 5% of the original. This gives one model with and without runtime curvature, differing only in that ingredient.

Finally, every trained model is exported through ONNX and TensorRT. We measure wall-clock latency on a Jetson-class device, following the joint-mapping paper's protocol. We also apply 8-bit and 4-bit post-training quantization using the analytical clipping method of Banner et al.. The quantization test matters because curved-space operations near the disk boundary are precision-sensitive. The hyperbolic networks survey identifies these operations as poorly matched to accelerator hardware.

Baselines

The naive reference point is the same Euclidean backbone with joints in default dataset index order and no structural channel. It shows how much any structure buys. For the shallow-network regime, the published floor is TSRJI: 73.3% on NTU60 cross-subject, 80.3% cross-view, and 67.9% on NTU120 cross-subject (TSRJI). We retrain it in our pipeline so that the comparison uses identical augmentation and epochs.

The strongest existing method on the exact question is HyLiFormer, because it is the only model that pays runtime curvature cost specifically to model skeletal hierarchy. Its reported results are 91.7% top-1 on NTU60 cross-subject and 87.5% on NTU120 cross-subject, at 1.9M parameters and 3.5G FLOPs (HyLiFormer). No reported result was found on Lacuna for HyLiFormer's on-device latency. We will retrain it with its published curvature setting and measure its latency ourselves on the same device and TensorRT version as every other model.

Two Euclidean efficiency baselines complete the comparison. The first is MM-GCN, a graph network that pools 25 joints into 5 hand-chosen parts. It reports 90.8% at 13.2 ms, against 14.1 ms for its CTR-GCN base. The second is FreqMixFormerV2, a frequency-domain transformer reporting 90.7% with 1.2M parameters.

All models get the same training data, the same 5-seed protocol, and an equal hyperparameter search budget of 20 trials each. The key ablation crosses three factors: layout (index, depth-first with duplicates, angular single-visit), structural channel (none, graph Laplacian eigenvectors, learned embeddings, hyperbolic coordinates), and runtime geometry (Euclidean, HyLiFormer-style hyperbolic).

We count it as beating HyLiFormer if our Euclidean model comes within 0.5 points of it on NTU60 and NTU120 cross-subject, averaged over seeds. It must do so while running at least 1.5 times faster on device, and while losing at least 1 point less accuracy under 4-bit quantization.

Why not just…?

Why not just use HyLiFormer as is? It may well be the most accurate option. But its own limitations section concedes conversion overhead, and FLOP counts do not capture the cost of transcendental functions that lack fused kernels. Matched on-device latency and quantization tests are exactly the measurements missing from the record. If HyLiFormer wins on accuracy and the latency gap turns out negligible, our hypothesis fails, and we will say so.

Why not just learn joint positional embeddings or a permutation end to end? This is the most serious threat. With only 25 joints, learned embeddings can in principle discover the hierarchy. The learned-embedding cell of the ablation tests this directly. We predict the fixed hyperbolic prior helps mainly where data is scarce: the 10%-label setting from HYSP and NTU120 cross-setup. If learned embeddings tie everywhere, we will report the conclusion that hierarchy is cheap to learn at this scale.

Why not just use Laplacian eigenvectors or plain tree distances instead of hyperbolic coordinates? For a small fixed tree, Euclidean spectral encodings are a fair substitute, and they may match ours. The ablation separates the two. If they match, the curvature-specific claim collapses to a structure claim. That would still answer the runtime question, which is the main point.

Why not just use a downsampling graph network like MM-GCN? It is already at 90.8% and 13.2 ms. We compare against it on the same device. Its fixed five-part pooling, however, discards finger and hand detail. We therefore additionally report accuracy on fine-grained confusable NTU pairs, such as reading versus writing, which TSRJI flags as a weakness of skeleton images.

Proposed Experiments

We evaluate on NTU RGB+D 60 under the cross-subject and cross-view protocols (NTU RGB+D), NTU RGB+D 120 under cross-subject and cross-setup, and NW-UCLA for view robustness. We report top-1 accuracy, parameters, FLOPs, TensorRT latency at batch size 1, and accuracy after 8-bit and 4-bit quantization. We also report accuracy on the ten most-confused class pairs and in a 10%-label training regime.

The first decisive check isolates curvature. In the matched HyLiFormer-architecture pair, the gap between runtime-hyperbolic and Euclidean-with-hyperbolic-layout should be under 0.5 points. The gap between runtime-hyperbolic and Euclidean-without-layout should be much larger. If both gaps are similar, the layout carries no hierarchical information.

The second check isolates ordering. In the shallow network, angular single-visit order should match or beat depth-first order with duplicates while using fewer rows. This replicates and extends TSRJI's 5.5-point ordering effect.

The third check, using the TREEREP co-motion tree, tests whether a data-derived hierarchy reduces fine-grained confusions relative to the anatomical tree.

The fourth check is the deployment one. Under 4-bit post-training quantization, the Euclidean model should degrade less than the hyperbolic one. That would tie the efficiency argument to a concrete, hardware-level difference rather than to FLOP counts alone.