Abstract
Skeleton-based action recognition is a critical task in computer vision, leveraging 3D joint trajectories to identify human activities. This paper presents two innovative modules designed to enhance recognition performance: Active Coordinate System Conversion (ACSC) and Spatio-Temporal-Unit Feature Enhancement (STUFE). ACSC adaptively learns optimal coordinate transformations for skeleton data alignment, while STUFE captures global co-occurrence relationships between joints across both spatial and temporal dimensions. Experimental results on the NTU-RGB+D and SBU Kinect Interaction datasets demonstrate that these modules achieve state-of-the-art accuracy and are compatible with various backbone architectures, including CNNs and GCNs.
Introduction
Human action recognition using skeleton data offers a compact and robust representation of movement, less sensitive to lighting conditions or background clutter than raw video. However, existing methods often struggle with two primary challenges. First, data alignment typically relies on fixed coordinate systems—usually Cartesian coordinates relative to the camera. While some researchers have explored fixed conversions to cylindrical coordinates, these transformations are rigid and do not adapt to the specific characteristics of different actions or datasets.
Second, feature learning in current models is often fragmented. Most architectures process spatial dependencies (which joints move together) and temporal dependencies (how a joint moves over time) in separate stages. This sequential or parallel processing often fails to capture the integrated global "co-occurrence" of joints—for instance, the critical relationship between a hand joint at the start of an action and a back joint at the end of a "touching back" movement.
Proposed Methodology
The proposed framework introduces a plug-and-play architecture that can be integrated into existing "backbone" models, such as Convolutional Neural Networks (CNNs) or Graph Convolutional Networks (GCNs). The pipeline consists of raw data input, coordinate conversion, feature extraction, feature enhancement, and final classification.
Active Coordinate System Conversion (ACSC)
Unlike traditional methods that use hard-coded mathematical transformations, ACSC employs a differentiable neural network, specifically a Multi-Layer Perceptron (MLP), to learn a custom coordinate system. This learnable approach allows the model to find the most effective alignment for skeleton data based on the training gradients.
A significant challenge with learnable transformations is ensuring they do not lead to a "singularity" or collapse of the coordinate system where information is lost. To mitigate this, the authors initialize the ACSC module as an identity matrix using a self-encoder strategy. This ensures that the initial state preserves all raw data information, which is then fine-tuned during the training process to optimize classification performance.
Spatio-Temporal-Unit Feature Enhancement (STUFE)
The STUFE module is designed to break the traditional silos of spatial and temporal processing. It treats every joint at every time step as an individual "unit." For a sequence with $J$ joints and $T$ frames, there are $J \times T$ such units. STUFE calculates a "distance" or correlation score between every possible pair of units across the entire sequence.
By utilizing an attention-like mechanism—specifically a weighted summation based on these correlation scores—STUFE enhances the feature map. This allows highly correlated joints to be processed together even if they are far apart in space or time. This "integrated and global" approach enables the model to learn complex co-occurrences that are often missed by localized spatio-temporal filters.
Experimental Results
The effectiveness of the proposed modules was evaluated on two gold-standard benchmarks in the field:
- NTU-RGB+D: This is currently the largest and most widely used dataset, featuring 56,000 samples across 60 action classes. The proposed model achieved 86.9% accuracy on the Cross-Subject (CS) protocol and 92.5% on the Cross-View (CV) protocol, outperforming previous heavyweights like STGR-GCN.
- SBU Kinect Interaction: Focused on two-person interactions like hugging or kicking, this dataset contains 282 sequences. The model reached a top-1 accuracy of 98.7%, nearly perfecting the benchmark.
Key Insights and Efficiency
- Learnable Preprocessing: The success of ACSC demonstrates that "one size does not fit all" in data preprocessing. Letting the model guide the coordinate transformation provides the flexibility needed for complex, multi-view actions.
- Backbone Agnostic: A major strength of these modules is their versatility. They act as "upgrades" that can be applied to both CNN and GCN architectures without requiring fundamental changes to the underlying model.
- Computational Trade-off: Despite the added complexity of global attention in STUFE, the training time only increased by approximately 13.37%. The authors argue this is a highly reasonable trade-off given the significant accuracy gains.
Conclusion
This research demonstrates that learnable coordinate transformations and global co-occurrence modeling significantly improve skeleton-based action recognition. While ACSC provides a flexible alignment strategy that adapts to the data, STUFE ensures that critical joint relationships are captured across the entire action sequence. Future work will focus on exploring more complex embedding models for the STUFE distance function and applying these global co-occurrence ideas to other modalities beyond skeletons, such as raw video or depth maps.
Summary updated