Hand Gesture Recognition in Complex Background Based on Convolutional Pose Machine and Fuzzy Gaussian Mixture Models

Tong Zhang, Huifeng Lin, Zhaojie Ju, Chenguang YangInt. J. Fuzzy Syst 2020 · DOI

Abstract

Vision-based hand gesture recognition often struggles in real-world environments characterized by cluttered backgrounds, varying lighting, and non-gesture inputs. This paper presents a robust two-stage framework designed to overcome these challenges. The first stage utilizes a multi-stage Convolutional Pose Machine (CPM) to accurately estimate 21 hand keypoints, leveraging spatial context to handle occlusions and environmental noise. The second stage employs a Fuzzy Gaussian Mixture Model (FGMM) for gesture classification. By incorporating fuzzy logic into the Expectation-Maximization algorithm, the model achieves faster convergence and improved performance on limited datasets. Experimental results demonstrate a high recognition accuracy of 98.06% and a 95% success rate in rejecting non-gesture patterns, all while maintaining real-time performance at 30 FPS.

Introduction

Traditional gesture recognition systems frequently rely on hand segmentation techniques, such as skin color or depth-based thresholding. While effective in controlled laboratory settings, these methods often fail in complex real-world scenarios where backgrounds are cluttered, lighting is inconsistent, or the user's arm is visible. Furthermore, many existing classifiers are "closed-set," meaning they attempt to categorize every hand shape into a predefined gesture class, leading to false positives for "non-gesture" movements like waving or scratching. This research addresses these limitations by proposing a system that is robust to environmental noise and capable of rejecting unknown patterns.

Methodology

The proposed framework consists of a two-stage pipeline: hand pose estimation followed by gesture classification.

Hand Pose Estimation (CPM)

Instead of pixel-level segmentation, the system uses a Convolutional Pose Machine (CPM) to locate 21 specific joints (keypoints). The CPM employs a sequential architecture where each stage refines the predictions of the previous one. By utilizing spatial context—the physical relationship between joints—the model can accurately predict keypoint locations even when fingers are partially occluded.

The multi-stage architecture of the Convolutional Pose Machine (CPM) used for keypoint localization.

Hand Pose Classification (FGMM)

Once keypoints are localized, their coordinates are normalized to ensure invariance to scale and position. These features are then processed by a Fuzzy Gaussian Mixture Model (FGMM). The "fuzzy" aspect involves incorporating fuzzy membership degrees into the Expectation-Maximization (EM) algorithm, allowing for faster convergence and better handling of limited training data. As a generative model, the FGMM calculates the likelihood of a set of keypoints belonging to a known gesture. If the likelihood falls below a predefined threshold, the input is rejected as a "non-gesture."

Experimental Results

The system was evaluated using the Rendered Hand Pose (RHD) dataset for CPM training and a custom dataset of seven static gestures for classification.

Performance and Refinement

The multi-stage nature of the CPM is critical for accuracy. As shown in the heatmap visualizations, the confidence of keypoint locations increases significantly through successive stages of the network, effectively filtering out background noise.

Visualization of heatmap refinement across the three stages of the CPM, showing improved localization accuracy.

Accuracy and Efficiency

The proposed method achieved an overall accuracy of 98.06%, significantly outperforming Support Vector Machines (91.11%) and Multi-Layer Perceptrons (88.33%). Notably, the FGMM demonstrated superior rejection capabilities, identifying "non-gesture" patterns with 95% accuracy, compared to 65% for SVMs. The entire pipeline runs in approximately 33ms on an NVIDIA 2080 GPU, enabling real-time interaction at 30 FPS.

Conclusion

By shifting from segmentation to keypoint-based pose estimation and utilizing fuzzy generative models for classification, this framework provides a robust solution for gesture recognition in unpredictable environments. Future work will focus on integrating temporal models like LSTMs to handle dynamic gestures and improving performance under extreme occlusions.

Summary updated

BibTeX
@article{zhang2020hand,
  author = {Tong Zhang and Huifeng Lin and Zhaojie Ju and Chenguang Yang},
  title = {{Hand Gesture Recognition in Complex Background Based on Convolutional Pose Machine and Fuzzy Gaussian Mixture Models}},
  journal = {Int. J. Fuzzy Syst.},
  year = {2020},
  doi = {10.1007/s40815-020-00825-w},
}
Figure