TL;DR
This paper introduces a novel action classification method using a discriminative, multi-level Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM). By sharing learned pose representations across action classes and incorporating a discriminative learning approach, the model achieves improved accuracy in recognizing human actions from depth sensor data.
What Problem Does This Paper Address?
Recognizing human actions in video sequences is a challenging task with applications in surveillance, human-computer interaction, and more. Depth sensors like Kinect provide 3D joint positions, which are a rich source of information for inferring actions. However, accurately modeling these actions is difficult due to variations in pose, viewpoint, and the need to distinguish between similar actions. Traditional Hidden Markov Models (HMMs) require pre-specifying the number of hidden states and often focus on explaining data within a class rather than discriminating between classes, leading to suboptimal classification. Non-parametric Bayesian methods like HDP-HMM can automatically infer the number of states, but standard implementations are generative and may not yield the best discriminative performance. This paper addresses the need for a more robust and accurate action classification system that can leverage shared information across actions and learn discriminative features.
Methods
The core of this work is a multi-level HDP-HMM framework. Unlike standard HDP-HMMs that might model each action class independently, this approach uses a hierarchical structure. A global HDP-HMM learns shared pose representations (mixture components) across all action classes. An additional, class-specific hierarchical level then transforms these shared parameters to account for the unique characteristics of each action. This allows for "borrowing" information from similar actions during training, improving the learning of pose representations.
Crucially, the authors introduce a discriminative learning component. Instead of solely relying on a generative model that explains the data, they incorporate a "discriminative term" into the learning objective. This term encourages the model to create larger margins between different action classes, leading to better separation and classification. To handle the complexity introduced by this discriminative term, they employ a normalized gamma process formulation for the HDP and use elliptical slice sampling for posterior inference, which is an efficient method for sampling from complex posterior distributions. The features used are derived from 3D joint positions, either as relative joint positions within a frame or as more complex HOG-based descriptors that incorporate temporal information.
Datasets
The proposed method is evaluated on two publicly available datasets:
- UTKinect-Action dataset: Contains actions like walk, sit-down, stand-up, pick-up, carry, throw, pull, wave, and clap-hands.
- MSR-Action3D dataset: A larger dataset with 20 actions, including variations of arm movements, punches, kicks, and more.
Both datasets provide depth images and corresponding 3D joint positions. The authors note that these datasets may contain noisy joint position estimates due to occlusions.
Key Insights and Ideas
- Shared Representations for Action Classes: The multi-level HDP-HMM effectively learns common pose structures across different actions, which is beneficial because many actions share similar underlying poses. This is a key insight for improving efficiency and robustness.
- Discriminative Learning for HMMs: Directly applying discriminative learning principles (like large margins) to HMMs, especially non-parametric ones like HDP-HMM, is challenging. The paper proposes a novel way to incorporate this into the HDP-HMM framework, leading to improved classification performance.
- Elliptical Slice Sampling for Complex Posteriors: The need for efficient sampling from the complex posterior distribution induced by the discriminative HDP-HMM is addressed by elliptical slice sampling, highlighting its utility in complex Bayesian models.
- Feature Engineering Matters: The paper explores two feature representations (relative joint positions and HOG-based descriptors), demonstrating that feature choice significantly impacts performance, especially for complex actions.
Key Results
- Improved Accuracy: The discriminative multi-level HDP-HMM significantly outperforms traditional parametric HMMs on both datasets.
- On UTKinect-Action, accuracy increased from 58.2% (parametric HMM) to 83.1% (discriminative multi-level HDP-HMM).
- On MSR-Action3D, accuracy rose from 48.7% (parametric HMM) to 83.3% (discriminative multi-level HDP-HMM).
- Generative vs. Discriminative: While sharing parameters in a multi-level HDP-HMM (generative learning) already improved accuracy over parametric HMMs, the addition of discriminative learning further boosted performance.
- Comparison to State-of-the-Art: The proposed method achieves competitive results, though it is outperformed by some recent methods on the UTKinect-Action dataset, which may be due to differences in evaluation protocols (e.g., cross-subject vs. leave-one-out cross-validation).
Limitations and Future Work
- Noisy Joint Positions: The paper acknowledges that the provided 3D joint positions can be noisy due to occlusions, which can affect classification accuracy.
- Feature Complexity: While the HOG-based features show good performance, they are more complex than simple relative joint positions.
- Future Work: The authors suggest broadening the discriminative criteria and applying the technique to classify activities involving humans and objects, indicating a desire to extend the model's capabilities. They also mention the potential for semi-supervised learning by incorporating unlabelled data.
Summary updated