TL;DR
This paper introduces an efficient, online algorithm for recognizing continuous human actions from Kinect skeletal data. It cleverly avoids explicit action start/end detection by segmenting data into "pose" and "motion" features and using a variable-length Maximum Entropy Markov Model (MEMM) for classification, achieving high accuracy on challenging datasets.
What Problem Does This Paper Address?
Recognizing human actions from video is crucial for applications like human-robot interaction and surveillance. However, real-world scenarios often involve continuous streams of data containing multiple, unsegmented actions. Existing methods often struggle with this by either requiring pre-segmented actions or using fixed-size sliding windows, which can be inefficient or miss action boundaries. This paper tackles the challenge of online continuous human action recognition (CHAR), where actions must be identified as they happen, without prior knowledge of their start and end times, using readily available skeletal data from depth sensors like the Kinect.
Methods
The proposed algorithm breaks down the CHAR problem into several key stages:
- Feature Extraction: It utilizes Normalized Relative Orientations (NROs) of human joints. NROs are computed relative to parent joints and are designed to be invariant to subject height, limb length, and distance to the camera, making them robust.
- Online Segmentation: This is a core innovation. Instead of relying on external cues, the algorithm dynamically segments the feature sequence into "pose feature segments" (representing still or slow-moving poses) and "motion feature segments" (representing significant movements). This segmentation is based on the "potential difference" of feature vectors, effectively identifying points of significant change in human pose and motion.
- Offline Model Training: For each human action, key poses are extracted using Gaussian Mixture Models (GMMs) on pose feature segments. Atomic motions, representing transitions between key poses, are then extracted from motion feature segments. This creates a model of each action as a sequence of key poses and atomic motions.
- Online Model Matching: As new segments are identified online, their likelihood of matching the pre-trained key poses and atomic motions is calculated.
- Classification with Variable-Length MEMM: This is another key contribution. A variable-length Maximum Entropy Markov Model (MEMM) is employed. Unlike traditional MEMMs that assume fixed-length feature sequences, this approach can adapt to varying segment lengths. This allows it to effectively leverage the discriminatory power of both significant (discriminatory) and less significant (neutral) poses/motions, improving both accuracy and efficiency. It also avoids the need to explicitly detect action start/end points.
Datasets
The paper evaluates its algorithm on two publicly available RGB-D datasets:
- Cornell CAD-60: This dataset contains 12 daily activities performed by 4 people in 5 different locations. It's noted for its focus on everyday activities.
- MSR Daily Activity 3D: This dataset includes 16 activities performed by 10 people in an indoor environment.
To simulate real-world continuous action scenarios, the authors combined test data sequences from these datasets into longer sequences containing multiple actions, without explicit segmentation marks.
Key Insights and Ideas
- Segmentation as a Key to Continuous Recognition: The paper demonstrates that by effectively segmenting continuous data into meaningful "pose" and "motion" segments, the problem of recognizing continuous actions can be simplified.
- Variable-Length MEMM for Adaptability: The use of a variable-length MEMM is crucial. It allows the model to be flexible in how it processes sequences, recognizing that some actions might be defined by a few key moments, while others require a longer temporal context. This contrasts with fixed-window approaches.
- NROs as Robust Features: The choice of Normalized Relative Orientations (NROs) as features proves effective, offering invariance to scale and translation, which are common challenges in human motion analysis.
- No Explicit Start/End Detection Needed: A significant advantage is that the algorithm doesn't require pre-detection of action boundaries, making it more practical for real-time applications.
Key Results
- CAD-60 Dataset: The proposed algorithm achieved an average recognition accuracy of 92.0% on the CAD-60 dataset. This performance is highlighted as remarkable, especially when compared to methods that operate on segmented data.
- MSR Daily Activity 3D Dataset: The accuracy on this dataset was 54.7%. While lower than on CAD-60, it's noted that this dataset presents greater challenges due to similar activities and the lack of human-object interaction features (which some competing methods utilize).
- Efficiency: The average processing time per frame is reported as approximately 0.1 ms, making it highly efficient for online applications.
Limitations and Future Work
- Skeletal Data Limitations: The authors acknowledge that skeletal data alone has limitations, especially for distinguishing very similar actions or those involving subtle human-object interactions.
- Feature Extraction Scope: The current work focuses solely on skeletal data. Future work could explore integrating RGB-D images and human-object interaction features for improved recognition.
- Hierarchical Models: The paper suggests that extracting features for individual limbs and training hierarchical models could be a promising direction for future research.
- Action Boundaries: While the algorithm doesn't require explicit boundary detection, the paper notes that misrecognition can occur at action boundaries, particularly when actions change suddenly or are very similar. The "Delayed Recognition" metric in Table 2 highlights this.
Summary updated