TL;DR
This paper introduces a novel hand gesture recognition system using a Kinect sensor, combining a new "FGFF" descriptor for fingertip and finger features with modified Hu moments for hand contour analysis. This approach aims to overcome the limitations of existing methods by providing more comprehensive and discriminative features, leading to improved accuracy and robustness in recognizing ten distinct hand gestures.
What Problem Does This Paper Address?
Hand gesture recognition is a challenging problem due to the inherent complexity and variability of human hand movements. Existing methods often struggle with insufficient or non-robust features, leading to poor performance and weak generalization. Traditional vision-based approaches are susceptible to environmental factors like lighting and background clutter, while 3D depth-based methods, though promising, can be computationally expensive or require specialized equipment like data gloves. The paper specifically targets the need for a comprehensive and discriminative feature representation that is robust to variations in hand size, position, and orientation, and can effectively capture both fine-grained finger details and overall hand shape.
Methods
The proposed method combines several novel techniques:
- Fingertip Gradient orientation with Finger Fourier (FGFF) Descriptor: This descriptor captures two key aspects:
- Fingertip Gradient Orientations: It extracts palm center, fingertips, and their gradient orientations, using relative positions to achieve translation and rotation invariance.
- Finger Fourier Descriptor: It applies a 1D Discrete Fourier Transform to the finger's contour points to extract frequency-domain features, focusing on lower frequencies that carry most of the gesture's energy.
- Modified Hu Moment Invariants: To capture the overall hand contour, the paper proposes a modification to the standard Hu moment invariants. By using lower exponents, they aim to improve robustness to noise and reduce computational complexity while maintaining translation, rotation, and scale invariance.
- Hand Segmentation and Finger Extraction: Two algorithms are introduced for segmenting the hand region from depth images using a double-threshold-based region growing method. This is followed by algorithms to extract the palm center, fingers, and fingertips.
- Weighted AdaBoost Classifier: A multi-class classifier is built by combining two base classifiers:
- Finger-Earth Mover's Distance (FEMD): Used to measure similarity between FGFF descriptors of gestures.
- Support Vector Machine (SVM): Used to classify gestures based on the modified Hu moment features. The weighted AdaBoost approach combines the strengths of both classifiers, giving more importance to the one with higher recognition accuracy.
Datasets
The paper utilizes a custom-collected dataset for training and evaluation.
- Custom Ten-Gesture Dataset: This dataset comprises depth images of ten hand gestures representing digital numbers from zero to nine. The data was collected from ten students performing gestures at three different distances (80 cm, 120 cm, 150 cm) and repeated 20 times per person, resulting in a total of 2000 samples.
Key Insights and Ideas
- Combining Local and Global Features is Crucial: The paper strongly advocates for the synergy between detailed fingertip/finger features (FGFF) and global hand contour features (modified Hu moments). This dual approach captures a richer representation of gestures than either alone.
- Modified Hu Moments for Robustness: The modification of Hu moments with lower exponents is a clever way to retain their invariance properties while making them more resilient to noise and computationally efficient, addressing a known limitation of traditional Hu moments.
- FGFF Descriptor Captures Nuance: The FGFF descriptor's ability to capture both the orientation of fingertips and the spectral characteristics of finger contours provides a powerful way to differentiate subtle variations in gestures.
- Weighted AdaBoost for Ensemble Power: Using weighted AdaBoost to combine FEMD and SVM effectively leverages the strengths of different feature types and classification strategies, leading to a more robust and accurate final decision.
- Double-Threshold Region Growing for Segmentation: The proposed segmentation method, while simple, effectively isolates the hand region from depth images, providing a clean input for subsequent feature extraction.
Key Results
- High Recognition Accuracy: The proposed algorithm achieved a mean accuracy of 96.6% on the challenging ten-gesture dataset.
- Efficiency: The system operates with an average processing time of 0.05 seconds per frame.
- Robustness to Distance: The algorithm demonstrated good performance across different distances from the Kinect sensor, with average accuracies above 94%.
- Comparison with Benchmarks: The proposed method outperformed three benchmark algorithms (Ren [23], Huang [24], and Pu Xingcheng [28]) in terms of recognition accuracy and stability. For instance, it achieved a higher average accuracy (96.6%) compared to Huang [24] (96.1%) and Ren [23] (95.0%).
- Discrimination of Confused Gestures: The paper shows that the combined approach effectively distinguishes between gestures that are often confused, such as gestures six, seven, and nine.
Limitations and Future Work
- Confused Gestures: While the algorithm shows improvement, some gestures (six, seven, and nine) are still prone to confusion, particularly due to variations in finger thickness and curvature.
- Training Data Volume: The paper notes that accuracy increases with the volume of training data, suggesting that larger datasets could further improve performance.
- Future Work: The authors suggest exploring more representative features, further improving robustness, developing interesting HCI applications, and deploying the system on mobile robots for routine housework tasks.
Summary updated