Abstract
Vision-based hand gesture recognition often struggles in real-world environments characterized by cluttered backgrounds, varying lighting, and non-gesture inputs. This paper presents a robust two-stage framework designed to overcome these challenges. The first stage utilizes a multi-stage Convolutional Pose Machine (CPM) to accurately estimate 21 hand keypoints, leveraging spatial context to handle occlusions and environmental noise. The second stage employs a Fuzzy Gaussian Mixture Model (FGMM) for gesture classification. By incorporating fuzzy logic into the Expectation-Maximization algorithm, the model achieves faster convergence and improved performance on limited datasets. Experimental results demonstrate a high recognition accuracy of 98.06% and a 95% success rate in rejecting non-gesture patterns, all while maintaining real-time performance at 30 FPS.
Introduction
Traditional gesture recognition systems frequently rely on hand segmentation techniques, such as skin color or depth-based thresholding. While effective in controlled laboratory settings, these methods often fail in complex real-world scenarios where backgrounds are cluttered, lighting is inconsistent, or the user's arm is visible. Furthermore, many existing classifiers are "closed-set," meaning they attempt to categorize every hand shape into a predefined gesture class, leading to false positives for "non-gesture" movements like waving or scratching. This research addresses these limitations by proposing a system that is robust to environmental noise and capable of rejecting unknown patterns.
Methodology
The proposed framework consists of a two-stage pipeline: hand pose estimation followed by gesture classification.
Hand Pose Estimation (CPM)
Instead of pixel-level segmentation, the system uses a Convolutional Pose Machine (CPM) to locate 21 specific joints (keypoints). The CPM employs a sequential architecture where each stage refines the predictions of the previous one. By utilizing spatial context—the physical relationship between joints—the model can accurately predict keypoint locations even when fingers are partially occluded.
Hand Pose Classification (FGMM)
Once keypoints are localized, their coordinates are normalized to ensure invariance to scale and position. These features are then processed by a Fuzzy Gaussian Mixture Model (FGMM). The "fuzzy" aspect involves incorporating fuzzy membership degrees into the Expectation-Maximization (EM) algorithm, allowing for faster convergence and better handling of limited training data. As a generative model, the FGMM calculates the likelihood of a set of keypoints belonging to a known gesture. If the likelihood falls below a predefined threshold, the input is rejected as a "non-gesture."
Experimental Results
The system was evaluated using the Rendered Hand Pose (RHD) dataset for CPM training and a custom dataset of seven static gestures for classification.
Performance and Refinement
The multi-stage nature of the CPM is critical for accuracy. As shown in the heatmap visualizations, the confidence of keypoint locations increases significantly through successive stages of the network, effectively filtering out background noise.
Accuracy and Efficiency
The proposed method achieved an overall accuracy of 98.06%, significantly outperforming Support Vector Machines (91.11%) and Multi-Layer Perceptrons (88.33%). Notably, the FGMM demonstrated superior rejection capabilities, identifying "non-gesture" patterns with 95% accuracy, compared to 65% for SVMs. The entire pipeline runs in approximately 33ms on an NVIDIA 2080 GPU, enabling real-time interaction at 30 FPS.
Conclusion
By shifting from segmentation to keypoint-based pose estimation and utilizing fuzzy generative models for classification, this framework provides a robust solution for gesture recognition in unpredictable environments. Future work will focus on integrating temporal models like LSTMs to handle dynamic gestures and improving performance under extreme occlusions.
Summary updated