Activity recognition using a supervised non-parametric hierarchical HMM

Natraj Raman, Stephen J. MaybankNeurocomputing 2016 · DOI

TL;DR

This paper introduces a novel non-parametric hierarchical Hidden Markov Model (H-HMM) for activity recognition from depth image sequences. By leveraging a hierarchical structure and a non-parametric Bayesian approach, the model automatically infers the number of states and enables information sharing across activities, leading to improved classification accuracy, especially in challenging scenarios.

What Problem Does This Paper Address?

Classifying human activities from video is a complex task with applications ranging from surveillance to human-computer interaction. Traditional methods often struggle with the inherent variability in human actions, the need for large labeled datasets, and the difficulty in modeling the hierarchical nature of activities (e.g., an activity composed of several actions, each with multiple poses). This paper specifically tackles the challenge of robustly recognizing human activities from depth image sequences, which provide rich 3D information about human poses, while addressing the limitations of fixed state models and the need for explicit feature engineering.

Methods

The core of this work is a non-parametric Hierarchical Hidden Markov Model (H-HMM). Here's a breakdown of the key components:

  • Hierarchical Structure: The model uses a two-level hierarchy. The top level models "actions" (coarser temporal segments), and the bottom level models "poses" (finer temporal segments). This structure allows for a more natural representation of activities as sequences of actions and poses, facilitating information sharing.
  • Non-Parametric Bayesian Approach (HDP): Instead of pre-defining the number of states (a common limitation of traditional HMMs), the authors employ a Hierarchical Dirichlet Process (HDP). This allows the model to automatically infer the optimal number of action and pose states from the data, a significant advantage.
  • Factorized Features: The model uses a factorized representation for poses, separating skeleton pose (derived from 3D joint positions) and object pose (derived from depth image patches around joints). This allows for flexibility in incorporating multiple data channels.
  • Supervised Classification (Multinomial Logistic Regression): To classify activities, the authors integrate multinomial logistic regression. This regression model learns the relationship between the inferred action states and the activity labels. This approach allows for parameter sharing across activity classes and influences the inference of action states, ensuring they are discriminative.
  • Inference: A block sampling approach, adapted from dynamic programming algorithms, is used for efficient inference, enabling the model to handle sequential data effectively.

Datasets

The proposed model was evaluated on three real-world datasets:

  • Cornell Activity Dataset: Contains 12 activities performed by four subjects, recorded in various household locations. Features include 3D joint positions and local depth image patterns.
  • UTKinect-Action Dataset: Features 10 actions performed by 10 subjects, recorded from different views. This dataset primarily uses skeleton-based features.
  • HDM05 Dataset: A motion capture dataset (without depth images) used to demonstrate the model's applicability to generic sequential data. It includes activities like badminton, dancing, and workout.

Key Insights and Ideas

  • Automatic State Discovery: The non-parametric HDP is crucial for automatically determining the number of action and pose states. This removes a significant manual tuning step and allows the model to adapt to the complexity of the data.
  • Hierarchical Structure for Efficiency: The hierarchical decomposition of activities into actions and poses simplifies the model and enables efficient information sharing. Coarser action states can be learned more robustly, even with limited data, and then used to inform the learning of finer pose states.
  • Regression for Discriminative Power: Integrating multinomial logistic regression with the generative H-HMM allows the model to learn action states that are not only descriptive of the observations but also discriminative for activity classification. This is a key enabler for achieving state-of-the-art performance.
  • Flexibility with Data Channels: The factorized representation of poses makes the model adaptable to incorporating various data modalities (e.g., skeleton, object, potentially other sensor data) without fundamentally altering the model structure.

Key Results

The proposed model achieved state-of-the-art or comparable results on the evaluated datasets:

  • Cornell Activity Dataset: The full model achieved 100% accuracy for the "seen subject" setting and 85.4% accuracy for the "new subject" setting. This significantly outperformed parametric HMM and H-HMM baselines.
  • UTKinect-Action Dataset: The full model achieved 87.9% accuracy for the "seen subject" setting and 84.0% for the "new subject" setting.
  • HDM05 Dataset: The full model achieved a high classification accuracy of 92.5%.
  • Semi-Supervised Learning: Incorporating unlabelled examples improved classification accuracy, demonstrating the model's utility in scenarios with limited labeled data.

Limitations and Future Work

  • Misclassification of Similar Actions: The model still misclassifies activities with very similar motion patterns (e.g., "sit down" vs. "pick up" in UTKinect, or "cooking" activities in Cornell).
  • Object Interaction: The authors suggest that explicitly detecting and incorporating objects that subjects interact with could further improve performance.
  • Subject Variability and Occlusions: The model does not explicitly handle cases where activities vary significantly between subjects, or situations involving occlusions or poses out of view. Addressing these would likely require increased model complexity.
  • Future Work: The authors intend to extend the model for classifying long-running activities involving multiple subjects. They also suggest exploring improved feature extraction and introducing new variables to handle complex scenarios.

Summary updated

BibTeX
@article{raman2016activity,
  author = {Natraj Raman and Stephen J. Maybank},
  title = {{Activity recognition using a supervised non-parametric hierarchical HMM}},
  journal = {Neurocomputing},
  year = {2016},
  doi = {10.1016/j.neucom.2016.03.024},
}
Figure