Abstract
The development of RGB-D sensors that have been widely applied in human motion collection is driving research in skeleton-based human action recognition. In recent works, most models based on the attention mechanism are proposed to assign the same weights to all body joints for spatiotemporal feature modeling. However, they fail to consider the fact that the differential contribution of joint points to the human movement, which is a challenge to obtain the high-level performance skeleton presentations. Therefore, in this article, we propose a novel sustained attention model based on the above fact, which adaptively assigns corresponding weights to all the body joints to extract the key skeleton part in the global input sequence. We design a two-stream network based on RNNs and CNNs and integrate the sustained attention mechanism into each subnetwork, in which both the body joint weights and the input frame weights are learned effectively and thus resulting in superior performance. Next, in the training process, the skeleton is randomly transformed to enhance the robustness of this model and reduce overfitting. A group of ablation studies and visualization analyses are conducted to prove the validity and robustness of the proposed model. Extensive experiments on four benchmark datasets included the challenging interaction datasets demonstrate that our proposed model outperforms recent state-of-the-art works.
Introduction
Skeleton-based action recognition has gained significant traction with the advent of affordable RGB-D sensors like the Microsoft Kinect. Unlike raw video, skeleton data provides a compact representation of human movement. However, a persistent challenge in this field is effectively modeling the differential importance of various body joints and temporal frames. Most existing attention-based models either treat all joints as equal or focus solely on local temporal windows, failing to capture the global context of an action.
In reality, human actions are defined by specific movements: "walking" is primarily about the legs and feet, while "putting on glasses" is about the hands and elbows. Furthermore, some frames in a video sequence are more "descriptive" of an action than others. Traditional models often fail to capture these differential contributions, leading to recognition errors—especially when actions look similar or involve complex interactions with objects.
The Proposed Dual-Stream Architecture
To address these limitations, this research introduces a discriminative dual-stream network integrated with a novel Sustained Attention (SA) mechanism. The architecture leverages the complementary strengths of Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs).
The model consists of two primary streams:
- SA-LSTM (Recurrent Stream): Designed to capture temporal dynamics, this stream utilizes Long Short-Term Memory units. It is uniquely structured into two parts to implement the sustained attention mechanism.
- SA-CNN (Convolutional Stream): Based on the AlexNet architecture, this stream focuses on spatial feature extraction. The third convolutional layer is modified into a dual-channel structure, where one channel performs standard convolution and the other implements an "Attention Model Mapping" to highlight critical spatial features.
The final classification is a weighted average of both streams, with the CNN stream given triple the weight of the LSTM stream, a configuration found empirically to be optimal.
Sustained Attention Mechanism
The core innovation is the Sustained Attention (SA) mechanism. Unlike traditional "soft attention" that operates on local segments, SA considers the global input sequence to calculate weights. This mimics human perception, where an observer often needs to see a full action sequence to identify which joints were most critical to the movement.
In the SA-LSTM subnetwork, the process is divided into two parts:
- Part 1 (Joint-level Attention): The model learns to assign weights to specific joints (e.g., wrists vs. knees) based on their relevance to the action across the entire sequence.
- Part 2 (Frame-level Attention): The model identifies "key frames" that are most descriptive of the action, effectively filtering out redundant or non-informative temporal data.
Experimental Results
The model was rigorously tested on four benchmark datasets: MSR-Action-3D, SBU Kinect Interaction, SYSU 3D Human-Object Interaction, and Northwestern-UCLA. To enhance robustness and prevent overfitting, the researchers employed a data augmentation strategy involving random rotations and translations of the skeleton data, which provided a consistent ~3% boost in accuracy.
The proposed model achieved state-of-the-art performance across all benchmarks:
- MSR-Action-3D: 95.6% accuracy, surpassing previous benchmarks like DSRF.
- Northwestern-UCLA: 93.7% accuracy, a significant 1.1% improvement over the previous best.
- SYSU 3D: 84.6% accuracy, showing strong performance in complex human-object interaction scenarios.
Visualization of attention maps confirmed that the model accurately "locks on" to relevant joints, such as marking knees and ankles in red during a "kicking" action and elbows/wrists during "putting on glasses."
Conclusion and Future Work
The Sustained Attention mechanism represents a significant step forward in skeleton-based action recognition by providing a global context for feature weighting. While the model shows exceptional performance, the authors noted that it still struggles with complex "two-step" actions that last a long time and can be broken into distinct phases. Future work will focus on exploring more sophisticated skeleton pre-processing techniques and utilizing even larger-scale interaction databases to further refine the model's discriminative power.
Summary updated