Abstract
Skeleton-based action recognition is a critical task in computer vision, offering robustness against environmental variations. However, existing Graph Convolutional Networks (GCNs) often fail to capture the subtle intra-frame joint differences and inter-frame motion dynamics essential for distinguishing complex actions. This paper introduces a novel framework that combines a directional convolution operation with a spatio-temporal difference descriptor based on Symmetric Positive Definite (SPD) matrices. By utilizing rotation matrices to preserve geometric scale and learning on Riemannian manifolds, the proposed method achieves state-of-the-art results across multiple benchmarks, including a 3% accuracy boost on the G3D-Gaming dataset.
Introduction
Human action recognition from skeleton data (3D joint positions) has gained popularity due to its resilience to lighting and background changes. While standard GCNs are effective at modeling joint dependencies, they possess a significant "blind spot": they struggle to identify intra-frame differences (static pose signatures) and inter-frame dynamics (joint displacements over time). Without these features, models often fail to distinguish between actions with similar joint configurations but different movement patterns.
Methodology
The proposed architecture, Di-StddNet, introduces three primary innovations to address these limitations:
1. Directional Convolution
Unlike standard convolutions that use scalar weights, this method partitions the 3D space around a joint into $b$ regions. It assigns unique 3D rotation matrices as weights based on the direction of a neighboring joint. This approach allows the network to learn spatial dependencies while preserving the physical scale and geometric orientation of the skeleton.
2. Spatio-Temporal Difference Descriptor
The model generates two types of statistical descriptors represented as Symmetric Positive Definite (SPD) matrices:
- Spatial Difference Matrix: Captures the unique static pattern of a skeleton by analyzing the variance between all joints within a single frame.
- Temporal Dynamic Matrix: Captures motion by calculating joint displacements between adjacent frames.
3. Deep SPD Metric Learning
Since SPD matrices reside on a curved Riemannian manifold, standard Euclidean layers are unsuitable. The authors developed a 2D fully connected layer for dimensionality reduction and a novel rectification function. This function prevents eigenvalues from becoming too small, ensuring numerical stability during the "log-map" operation required to flatten the manifold for final classification.
Experimental Results
The Di-StddNet was evaluated on three standard benchmarks, consistently outperforming existing GCN and manifold-based methods:
- G3D-Gaming: Achieved 92.03% accuracy, a ~3% improvement over the previous state-of-the-art (LieNet).
- HDM05: Reached 82.32%, surpassing the strong ST-GCN baseline.
- Northwestern-UCLA: Achieved 93.8%, outperforming the AGC-LSTM model.
Ablation studies confirmed that both the directional convolution and the non-linear SPD mapping blocks were essential for these performance gains.
Conclusion and Future Work
By focusing on the differences between joints rather than just their absolute positions, and by respecting the geometric nature of the human body through rotation matrices, this framework sets a new standard for skeleton-based recognition. Future research may address the computational complexity of SPD manifold operations and extend the temporal descriptors to capture long-range dynamics beyond adjacent frames.
Summary updated