Abstract
Skeleton data, which consists of only the 2D/3D coordinates of the human joints, has been widely studied for human action recognition. Existing methods take the semantics as prior knowledge to group human joints and draw correlations according to their spatial locations, which we call the semantic perspective for skeleton modeling. In this paper, in contrast to previous approaches, we propose to model skeletons from a novel spatial perspective, from which the model takes the spatial location as prior knowledge to group human joints and mines the discriminative patterns of local areas in a hierarchical manner. The two perspectives are orthogonal and complementary to each other; and by fusing them in a unified framework, our method achieves a more comprehensive understanding of the skeleton data. Besides, we customized two networks for the two perspectives. From the semantic perspective, we propose a Transformer-like network that is expert in modeling joint correlations, and present three effective techniques to adapt it for skeleton data. From the spatial perspective, we transform the skeleton data into the sparse format for efficient feature extraction and present two types of sparse convolutional networks for sparse skeleton modeling. Extensive experiments are conducted on three challenging datasets for skeleton-based human action/gesture recognition, namely, NTU-60, NTU-120 and SHREC, where our method achieves state-of-the-art performance.
Introduction: Beyond Anatomical Priors
Skeleton-based action recognition has traditionally relied on "semantic" priors. Most models, such as Graph Convolutional Networks (GCNs), group joints based on biological connections—linking the hand to the elbow, for instance, because they are physically connected. While effective for identifying specific joint movements, this approach often overlooks critical spatial proximity.
Consider the action of "taking a selfie." In this movement, the hand moves close to the face. A purely semantic model might struggle to capture this relationship because the hand and head are distant in the skeletal graph. To address this, the authors of SC4D propose a dual-perspective framework that considers both the anatomical "What" (semantics) and the coordinate-based "Where" (spatial locations).
The SC4D Framework
The proposed framework consists of two specialized streams: SEM-Net and SPA-Net. These networks operate in parallel to extract complementary features from the skeleton data.
1. SEM-Net: Refined Semantic Modeling
SEM-Net is a Transformer-based model optimized for skeletal structures. Standard Transformers are permutation-invariant, meaning they do not inherently recognize that a "head" joint differs from a "foot" joint. To solve this, the authors introduced three key techniques:
- Cross-Frame Self-Attention (CFSA): An efficient attention mechanism that models correlations across frames without the computational burden of full spatio-temporal attention.
- Global Regularization (GR): A learnable bias added to the attention map that encodes fixed anatomical relationships, ensuring the model respects the invariant structure of the human body.
- Multi-Scale Temporal Feed-Forward (MS-TFF): Replaces standard feed-forward layers with multi-branch temporal convolutions to capture dynamics across various time scales.
2. SPA-Net: Sparse 4D Spatial Modeling
SPA-Net treats the skeleton as a sparse point cloud in 4D space (3D coordinates + time). Because skeleton data is extremely sparse—most of a 3D grid is empty space—applying standard dense convolutions is computationally wasteful.
SPA-Net utilizes sparse 3D and 4D convolutions to "tour" local areas of the coordinate space. This allows the network to identify discriminative patterns based on where joints are located in 3D space, regardless of their anatomical names. The authors found that direct 4D convolutions are particularly effective at modeling complex spatio-temporal dependencies compared to decomposed (3D+1D) versions.
Experimental Results
The SC4D model was evaluated on three major benchmarks, achieving state-of-the-art (SOTA) results across the board:
- NTU RGB+D 60: Reached 92.6% (Cross-Subject) and 97.1% (Cross-View) accuracy.
- NTU RGB+D 120: Achieved 88.7% (Cross-Subject) accuracy on one of the field's most difficult datasets.
- SHREC: Demonstrated generalizability by achieving 96.2% on the 14-gesture task and 94.4% on the 28-gesture task for hand gesture recognition.
Ablation studies confirmed the value of the dual-perspective approach. Adding SPA-Net to SEM-Net provided a consistent performance boost (e.g., +0.8% on NTU-60), proving that spatial proximity captures vital information that anatomical graphs miss.
Conclusion
SC4D represents a significant shift in skeleton-based action recognition by treating spatial location as a first-class prior alongside human anatomy. By combining a refined Transformer (SEM-Net) with efficient sparse 4D convolutions (SPA-Net), the framework achieves a more comprehensive understanding of human movement. Future work may involve exploring "early fusion" strategies to allow these two orthogonal perspectives to interact more deeply during the feature extraction process.
Summary updated