Real-Time Human Pose Estimation and Gesture Recognition from Depth Images Using Superpixels and SVM Classifier

Hanguen Kim, Sangwon Lee, Dongsung Lee, Soonmin Choi, Jinsun Ju, Hyun Myung2015 · DOI

TL;DR

This paper introduces a novel real-time human pose estimation and gesture recognition system that relies solely on depth information, making it robust to lighting variations and suitable for low-cost platforms. By leveraging superpixels for efficient body part classification and a keyframe extraction method for gesture recognition, the system achieves competitive performance without requiring prior human body models or GPU acceleration.

What Problem Does This Paper Address?

Estimating human pose and recognizing gestures are crucial for applications in human-computer interaction, robotics, and surveillance. While depth sensors like Kinect have democratized access to 3D data, accurately estimating human poses from noisy, low-resolution depth images remains challenging. Many existing methods rely on GPUs for real-time performance or require extensive calibration and training data, limiting their applicability on resource-constrained embedded systems. Furthermore, gesture recognition often struggles with variations in motion speed and environmental changes. This paper aims to bridge this gap by proposing a robust, real-time system that operates on CPUs, uses only depth data, and requires minimal calibration.

Methods

The proposed system comprises two main components: human pose estimation and gesture recognition.

Human Pose Estimation: * Superpixel-based Classification: Instead of classifying every pixel, the method first generates superpixels on the human body using the SLIC algorithm. These superpixels, which group perceptually similar regions, are then classified into body parts using a Support Vector Machine (SVM). This significantly reduces computation time. * Coordinate Conversion and Normalization: To handle variations due to sensor distance and human pose, superpixel coordinates are normalized and translated to the human body's center of mass. A scale factor derived from depth values is applied to account for distance changes. * SVM Classification and Optimization: A trained SVM classifier assigns each superpixel to a body part. A post-processing optimization step removes misclassified superpixels by measuring their distance to the mean of their assigned body part cluster. * Hand Tracking: For occluded hands, a Kalman filter-based hand tracker is employed to estimate their positions, ensuring continuous tracking even when direct classification is not possible.

Gesture Recognition: * Keyframe Extraction: To make gesture recognition robust to motion speed, keyframes are extracted from the pose information. A frame is considered a keyframe if the difference in joint positions from the previous keyframe exceeds a predefined threshold. This process is repeated until the current keyframe is similar to a previously extracted one. * Action Sequence Matching: Extracted keyframes form an "action sequence." This sequence is then compared against registered gesture action sequences. The system identifies the start keyframe of a registered gesture that best matches the first keyframe of the input sequence using the mean Mahalanobis distance. * Mahalanobis Distance: This metric is used to compute the comparison error between input and registered gestures, considering both the mean and standard deviation of joint positions. * Sliding Window Filter: A sliding window filter is applied to minimize misrecognition by ensuring a consistent gesture estimation over a short temporal window.

Datasets

The authors generated their own datasets for evaluation. These datasets consist of 30 video files captured by an ASUS Xtion Pro depth sensor at a height of approximately 2 meters, with images of size 320x240. The datasets contain approximately 11,000 frames in total. Ground truth data, including joint positions, were created using the iPi Mocap Studio, a markerless motion capture system providing centimeter-level accuracy. For gesture recognition, 50 test datasets were created for each of the five defined gestures, encompassing various body types, genders, and clothing.

Key Insights and Ideas

  • Depth-Only Robustness: The reliance on depth information alone makes the system inherently robust to lighting conditions, a significant advantage over RGB-based methods.
  • Superpixels for Efficiency: Using superpixels instead of per-pixel classification is a clever way to drastically reduce the computational load for body part identification, enabling CPU-only operation.
  • Keyframe Extraction for Speed Invariance: The keyframe extraction method effectively decouples gesture recognition from the speed at which a gesture is performed, a common challenge in action recognition.
  • Low-Cost Platform Viability: The entire system is designed to run on a CPU, making it suitable for deployment on embedded boards and mobile platforms, broadening its practical applications.
  • No Prior Model Required: The pose estimation method does not rely on a predefined human body model, allowing it to work with any subject without prior calibration.

Key Results

  • Pose Estimation Performance: The proposed method achieved an average error of 92.3 mm and 73.3 mm (standard deviation) for pose estimation. While OpenNI NiTE showed slightly better accuracy (64.3 mm mean error), the proposed method's advantage lies in its lack of a calibration step and faster initial pose estimation (67.0 ms vs. 2413.1 ms for OpenNI NiTE).
  • Computational Efficiency: The proposed pose estimation method requires approximately 0.81 GFlops per frame, which is about 56% of the computational cost of Shotton's algorithm (1.44 GFlops).
  • Gesture Recognition Accuracy: The system achieved an overall gesture recognition rate of 92.8%, with individual gesture recognition rates ranging from 86.0% ("Emergency") to 98.0% ("Request for emergency supplies").
  • Real-Time Operation: The system can achieve a total frame rate of 15 fps, with pose estimation taking 65 ms and gesture recognition taking less than 1 ms per frame.

Limitations and Future Work

  • Gesture Overlap: The paper notes that the "Request for emergency supplies" gesture had a higher false recognition rate, potentially due to motion similarities with other gestures. Future work could involve defining gestures with minimal overlap.
  • Viewing Angle Sensitivity: The assumption that gestures are performed within a ±30-degree range of the sensor might limit recognition at wider viewing angles. The authors suggest using multiple sensors to mitigate this.
  • Future Improvements: The authors suggest several avenues for future research:
  • Employing other classification algorithms, such as deep learning, for body part classification.
  • Incorporating kinematic constraints of the human body into pose estimation.
  • Estimating human orientation to compensate for orientation changes during pose estimation.
  • Utilizing better feature information for body part classification.

Summary updated

BibTeX
@misc{kim2015real,
  author = {Hanguen Kim and Sangwon Lee and Dongsung Lee and Soonmin Choi and Jinsun Ju and Hyun Myung},
  title = {{Real-Time Human Pose Estimation and Gesture Recognition from Depth Images Using Superpixels and SVM Classifier}},
  year = {2015},
  doi = {10.3390/s150612410},
}
Figure