HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition

Yue Li, Haoxuan Qu, Mengyuan Liu, Jun Liu, Yujun CaiarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces HyLiFormer, a novel approach for skeleton-based human action recognition that tackles the computational bottleneck of traditional Transformers. By leveraging hyperbolic geometry and linear attention, HyLiFormer efficiently models the hierarchical structure of skeleton data while significantly reducing computational complexity, making it suitable for real-world applications.

What Problem Does This Paper Address?

Skeleton-based human action recognition (HAR) is crucial for understanding human behavior in various applications like surveillance and human-computer interaction. While Transformer models have shown great promise in capturing long-range dependencies in sequential data, their quadratic computational complexity (O(N²)) with respect to sequence length (N) makes them impractical for real-time or resource-constrained scenarios. Existing linear attention mechanisms offer a computational advantage (O(N)) but struggle to effectively capture the inherent hierarchical structure of skeleton data, which is more akin to a tree than a linear sequence. Hyperbolic geometry, particularly the Poincaré model, is well-suited for modeling such hierarchical structures due to its exponential volume growth. However, mainstream linear attention mechanisms lack well-defined operations in hyperbolic space, creating a gap that HyLiFormer aims to bridge.

Methods

HyLiFormer introduces two key novel components:

  • Hyperbolic Transformation with Curvatures (HTC) module: This module maps skeleton data from Euclidean space into the Poincaré model of hyperbolic space. It involves three steps:
  • Unit Vector Calculation: Preserves directional information of joint movements.
  • Hierarchy Modeling: Uses the hyperbolic tangent (tanh) function to map magnitudes, effectively encoding hierarchical levels. Smaller magnitudes map closer to the origin (global features), while larger ones map towards the boundary (local features).
  • Scaling for Hyperbolic Constraints: Ensures the transformed data remains within the Poincaré model's unit ball.
  • Hyperbolic Linear Attention (HLA) module: This module performs self-attention operations directly within the hyperbolic space. It reformulates the traditional attention mechanism to avoid the explicit computation of the QKT similarity matrix, reducing complexity from O(N²) to O(NF²). This is achieved by reordering operations and aggregating key-value pairs into a fixed-size representation that the query interacts with. The softmax function is used as the kernel for similarity, ensuring numerical stability and preserving hyperbolic properties.

The overall framework projects input skeleton data into hyperbolic space using HTC, processes it with HLA, and then maps it back to Euclidean space using the inverse of HTC.

Datasets

The paper evaluates HyLiFormer on two widely-used datasets for skeleton-based human action recognition:

  • NTU RGB+D 60: Contains 60 action categories and 56,880 samples.
  • NTU RGB+D 120: An extended version with 120 action categories and over 114,000 samples.

Both datasets provide multi-modal information, but the study focuses solely on the skeleton joint unimodality. Evaluation is performed using standard cross-subject (X-Sub) and cross-view (X-View) testing protocols.

Key Insights and Ideas

  • Bridging Hyperbolic Geometry and Linear Attention: The core innovation is the successful integration of linear attention mechanisms within the hyperbolic Poincaré model for HAR. This addresses the limitations of both traditional Transformers (complexity) and standard linear attention (hierarchical modeling).
  • Hierarchical Structure is Key: The paper emphasizes that skeleton data's inherent hierarchical nature is crucial for effective action recognition. Hyperbolic geometry naturally captures this, and HyLiFormer's HTC module explicitly encodes it.
  • Efficiency Without Sacrificing Accuracy: The primary goal is to achieve significant computational efficiency gains without a substantial drop in recognition accuracy. HyLiFormer demonstrates this by reducing complexity while maintaining competitive performance.
  • Curvature Matters: Experiments show that the choice of curvature for the Poincaré model significantly impacts performance, with -1 yielding optimal results for this task.

Key Results

  • Computational Efficiency: HyLiFormer achieves a training time of 3.7 min/epoch on the NTU RGB+D 120 dataset, which is significantly faster than other transformer-based methods like ST-TR (19.4 min/epoch) and Zoom Transformer (4.8 min/epoch). It also boasts a low parameter count (1.9M) and FLOPS (3.5G).
  • Competitive Accuracy: On the NTU RGB+D 60 dataset (X-Sub60), HyLiFormer achieves an accuracy of 91.7%, which is competitive with state-of-the-art methods like SkateFormer (92.6%) and FreqMixFormer (91.5%). On NTU RGB+D 120 (X-Sub120), it achieves 87.5% accuracy.
  • Ablation Studies:
    • Directly applying existing linear attention mechanisms (RWKV, Mamba) to skeleton data showed unsatisfactory performance compared to HyLiFormer.
  • Applying existing linear attention mechanisms to hyperbolic space (by mapping back and forth) resulted in a slight accuracy improvement but a significant increase in training time, highlighting the inefficiency of this approach.
  • The curvature of the Poincaré model was found to be critical, with -1 yielding the best accuracy.

Limitations and Future Work

  • Limitations:
    • While HyLiFormer significantly reduces complexity, it still relies on mapping data back and forth between Euclidean and hyperbolic spaces, which can introduce some overhead.
  • The paper focuses on the skeleton unimodal data; integrating other modalities could further enhance performance.
  • Future Work:
    • Exploring more efficient ways to perform operations directly in hyperbolic space to further reduce computational overhead.
  • Investigating the application of HyLiFormer to other hierarchical data structures beyond skeleton data.
  • Extending the model to incorporate multi-modal information for improved action recognition.

Summary updated

Abstract

Transformers have demonstrated remarkable performance in skeleton-based human action recognition, yet their quadratic computational complexity remains a bottleneck for real-world applications. To mitigate this, linear attention mechanisms have been explored but struggle to capture the hierarchical structure of skeleton data. Meanwhile, the Poincar\'e model, as a typical hyperbolic geometry, offers a powerful framework for modeling hierarchical structures but lacks well-defined operations for existing mainstream linear attention. In this paper, we propose HyLiFormer, a novel hyperbolic linear attention Transformer tailored for skeleton-based action recognition. Our approach incorporates a Hyperbolic Transformation with Curvatures (HTC) module to map skeleton data into hyperbolic space and a Hyperbolic Linear Attention (HLA) module for efficient long-range dependency modeling. Theoretical analysis and extensive experiments on NTU RGB+D and NTU RGB+D 120 datasets demonstrate that HyLiFormer significantly reduces computational complexity while preserving model accuracy, making it a promising solution for efficiency-critical applications.
BibTeX
@misc{li2025hyliformer,
  author = {Yue Li and Haoxuan Qu and Mengyuan Liu and Jun Liu and Yujun Cai},
  title = {{HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition}},
  year = {2025},
  eprint = {2502.05869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.05869},
}
Figure