Abstract
Human action recognition is an important yet challenging task. In this paper, we propose a robust and effective framework to largely improve the performance of human action recognition using depth maps. The key contribution is the Motion History Image (MHI) and Static History Image (SHI) is used to represent depth sequence. And we optimize the condition to construct the MHI and SHI; it allows us to capture more critical information. The local binary pattern (LBP) are then computed to gain the compact feature representation of an action. We evaluate the proposed framework on MSR Action3D dataset. Experimental results indicate that the proposed approach outperforms the state-of-the-art methods and demonstrate the effectiveness of the proposed approaches.
Introduction
Recognizing human actions in video is a cornerstone of computer vision, with applications ranging from healthcare monitoring to human-computer interaction. Traditionally, researchers used standard RGB cameras, but these are plagued by sensitivity to lighting changes, background clutter, and occlusions. The advent of depth sensors (like the Microsoft Kinect) solved many of these issues by providing 3D structural information. However, new challenges emerged: depth maps often contain noise, many actions look nearly identical, and processing full 3D video sequences is computationally expensive.
Proposed Methodology
The proposed framework addresses these challenges by creating a feature representation that is both discriminative and computationally efficient. The pipeline consists of four main stages.
Multi-View Projection and Temporal Templates
To capture the 3D nature of movement, depth maps are projected onto three orthogonal planes: front, side, and top views. From these projections, two types of temporal templates are generated:
- Motion History Image (MHI): Captures where and how motion occurs by stacking consecutive frames into a single grayscale image where brighter pixels represent more recent movement.
- Static History Image (SHI): Preserves the static parts of the body (posture) during an action, providing critical context that motion alone might miss.
The authors optimized the thresholds ($\epsilon_m$ and $\epsilon_s$) used to construct these images to filter out noise caused by camera shake or minor body wobbles.
Feature Extraction with Local Binary Patterns (LBP)
The core of the feature extraction process involves applying Local Binary Patterns (LBP) to the MHI and SHI templates. LBP is a powerful texture descriptor that labels pixels by comparing them to their neighbors, effectively capturing the "texture" of the motion and posture.
Following feature extraction, Principal Component Analysis (PCA) is used to compress the feature vector while retaining 90% of the variance. Final classification is performed using a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel.
Experimental Results
The framework was evaluated on the MSR Action3D dataset, which contains 20 action types performed by 10 subjects. The model was tested under three protocols: Test One (1/3 training), Test Two (2/3 training), and the Cross Subject Test.
The results demonstrate significant improvements over previous methods:
- Perfect Accuracy: On "Test Two," the model achieved 100% accuracy across all action subsets.
- Cross-Subject Performance: On the challenging Cross Subject Test, the model reached 95.2% accuracy, outperforming the previous state-of-the-art (3DMTM-PHOG) at 90.7%.
- Discriminative Power: In "Action Set 2," which includes highly similar motions like drawing and punching, the model achieved a 91.2% recognition rate, surpassing other methods by nearly 10%.
Conclusion and Future Work
A key insight of this research is that static posture information (SHI) is as vital as motion information (MHI) for distinguishing between similar actions. By combining these templates with LBP, the system becomes resilient to the noise common in depth data. While the current method relies on fixed bounding boxes and specific view projections, future work will explore optimizing LBP parameters and multi-view fusion techniques to further enhance robustness in unconstrained environments.
Summary updated