TL;DR
This paper introduces a novel four-stream deep learning framework for action recognition in small-scale depth video datasets. By combining raw depth frames with three types of temporal motion frames, and leveraging DenseNet121 for feature extraction, followed by BLSTM and MHSA for sequence modeling, the approach achieves state-of-the-art performance even with limited training data.
What Problem Does This Paper Address?
Human Action Recognition (HAR) is a crucial area in computer vision with applications ranging from surveillance to human-computer interaction. While RGB-based methods have advanced significantly, they are susceptible to illumination variations and occlusions. Depth sensors offer a more robust alternative, but training deep learning models for action recognition using depth data is challenging due to the scarcity of large-scale datasets. Existing depth-based methods often rely on frame-level classification or struggle with limited training samples. This paper aims to bridge this gap by proposing a deep sequence learning framework specifically designed for small-scale depth video datasets, directly classifying sequences rather than individual frames.
Methods
The proposed framework utilizes a "four-stream" representation of the input depth video. This involves:
- Depth Frame Sequence (DFS): The original sequence of depth frames.
- Three Temporal Motion Frame Sequences (TMFS): These are derived from the DFS by applying a sliding window to extract chunks of frames. For each chunk, motion is captured by calculating differences between projected frames onto the xy, yz, and xz planes, creating three distinct motion sequences.
These four streams are then processed independently for feature extraction using a pre-trained DenseNet121 model. DenseNet121 is chosen for its ability to alleviate vanishing gradients and reuse features efficiently. The extracted frame-level features from each stream are then fed into separate Bi-directional Long Short-Term Memory (BLSTM) networks. BLSTMs are used to model temporal dependencies and correlations between consecutive frames. To further enhance the model's ability to capture important features, Multi-Head Self-Attention (MHSA) mechanisms are applied to the outputs of the BLSTM layers. Finally, the outputs from the four MHSA layers are flattened, concatenated, and passed through dense layers with batch normalization and dropout for action classification.
Datasets
The proposed framework is evaluated on two public small-scale depth video datasets:
- MSRAction3D: Contains 557 depth frame sequences across 20 action classes.
- DHA: Consists of 483 depth frame sequences across 23 action classes.
The paper follows standard train/test splits for these datasets, with performers of odd indices used for training and even indices for testing.
Key Insights and Ideas
- Four-Stream Representation: The core idea of using a combination of raw depth frames and multiple temporal motion views significantly boosts performance, especially when data is scarce. This suggests that diverse representations of motion are crucial for robust action recognition.
- BLSTM + MHSA Synergy: The combination of BLSTM for temporal modeling and MHSA for feature weighting proves effective. MHSA helps to emphasize the most discriminative frames within a sequence, improving the overall classification.
- Depth Data Robustness: The paper highlights the inherent advantages of depth data over RGB for action recognition, particularly its invariance to illumination changes and its ability to preserve 3D structure.
- Small-Scale Data Efficacy: The framework demonstrates that deep sequence learning can be successfully applied to small-scale depth datasets, offering a viable alternative to data augmentation.
Key Results
- MSRAction3D: The proposed method achieved an accuracy of 96%, outperforming the state-of-the-art 3D CNN-based method [29] by 1.9%. It also uses fewer video frames and trainable parameters.
- DHA: The framework achieved an accuracy of 95.2%, outperforming the recent 3D CNN-based method [29] by 2.3%.
- Ablation Studies: The four-stream approach significantly outperformed single-stream and three-stream models, demonstrating the effectiveness of combining raw depth and motion information. The order of BLSTM and MHSA also mattered, with MHSA after BLSTM yielding better results.
- Architecture Comparison: DenseNet121, despite using fewer features than DenseNet169 and ResNet101V2, achieved the best accuracy among the tested architectures.
Limitations and Future Work
- Confusion in Similar Actions: The system, like many others, exhibits confusion in distinguishing between action classes with very similar motion cues. For example, on MSRAction3D, "draw circle" and "draw tick" were sometimes confused.
- Limited Data Augmentation: While the four-stream approach mitigates the need for extensive data augmentation, the paper acknowledges that other techniques like spatial data augmentation, diverse frame number temporal augmentation, and advanced parameter tuning could further improve accuracy.
- Future Work: The authors suggest exploring these alternative augmentation strategies and parameter tuning techniques. They also believe their lightweight and effective architecture can be a valuable alternative for other small-scale depth dataset problems.
Summary updated