TL;DR
This paper introduces a real-time, vision-based system for interpreting 2D dynamic hand gestures to enable natural human-robot communication with a pet robot named Yuppy. The system segments hands using motion and color, extracts trajectory features, and translates them into robot commands, achieving over 90% accuracy for primitive gestures in unstructured environments.
What Problem Does This Paper Address?
As robots become more integrated into our daily lives, particularly in roles like "pet robots," the need for intuitive and natural interaction methods becomes paramount. Current human-robot interaction often relies on specialized input devices like keyboards or mice, which can be cumbersome and unnatural. While speech is one natural interface, humans also heavily rely on gestures and body language to convey intentions. This paper addresses the challenge of enabling robots to understand human hand gestures in complex, real-world environments, moving beyond controlled laboratory settings and specialized equipment. The goal is to create a more seamless and expressive communication channel between humans and robots, akin to how we interact with each other.
Methods
The proposed system employs a three-phase strategy for gesture interpretation:
- Hand Segmentation: This is a critical step for robust gesture recognition. The system uses a combination of motion and color information to segment the gesturing hand from cluttered backgrounds.
- Motion Detection: It utilizes a difference of three sequential images (It-2, It-1, It) to detect moving components. This is computationally efficient.
- Color Detection: The HLS (Hue, Lightness, Saturation) color model is used, specifically focusing on hue and saturation bounds derived from an a priori skin color model. This approach is more robust to illumination changes than simple RGB thresholding.
- Integration: Motion detection is applied first to constrain the search for skin color. A bounding box is computed around moving components, and then skin-colored regions within this box are identified. A function combining motion and skin-color detection weights is used to segment the hand, prioritizing regions with the highest score (largest area and most displaced pixels). The hand's centroid is then tracked in real-time.
- Feature Extraction: Once the hand is segmented and tracked, its trajectory information is stored in a feature vector.
- Gesture Segmentation: The system assumes a gesture begins when the hand's motion exceeds a certain velocity. A gesture ends when the hand stops moving for a short period (2-3 seconds).
- Trajectory Representation: The vertical (dy) and horizontal (dx) displacements of the hand's centroid are calculated and stored in a feature vector until the gesture ends.
- Gesture Recognition: The feature vector is then used to interpret the gesture.
- Primitive Gestures: The system defines 8 basic linear gestures (vertical, horizontal, diagonal) and circular gestures. For linear gestures, the (dx, dy) displacements cluster around specific axes, and their direction is determined by the side of the axis. For circular gestures, the centroid of displacements coinciding with the origin indicates circularity, and the direction (clockwise/counterclockwise) is deduced from the time sequence of displacements.
- Model-Based Matching: Identified primitive gestures are represented by a descriptor. This descriptor is then queried against a database of model gestures, each associated with a specific command for the robot.
- Composite Gestures: The system supports combining primitives into composite gestures by using a shorter pause between primitives than between distinct gestures. The descriptor for a composite gesture contains a sequence of primitive labels.
Datasets
Not explicitly discussed in the paper. The experiments involved 14 subjects with diverse skin tones performing a sequence of 16 gestures in a crowded lab environment. The system was tested on Yuppy, a pet robot prototype, using an external microcamera.
Key Insights and Ideas
- Leveraging Multiple Cues for Robust Segmentation: Combining motion and color detection significantly improves hand segmentation accuracy and robustness in complex environments, overcoming limitations of using either cue alone.
- HLS Color Space for Illumination Invariance: Using the HLS color model, particularly by disregarding lightness, provides better robustness to varying lighting conditions compared to simpler color spaces.
- Feature Vector as a Trajectory Representation: Storing the hand's 2D displacement trajectory provides a rich representation that can distinguish between various linear and circular gestures.
- Extensible System Design: The architecture allows for the addition of new primitive gestures and the creation of composite gestures, making the system adaptable to a growing lexicon of commands.
- Real-time Performance on Commodity Hardware: The system achieves real-time performance (15 frames per second) on a standard Pentium Pro workstation, demonstrating the feasibility of vision-based gesture interaction without specialized hardware.
Key Results
- Accuracy: The system achieved an accuracy rate of above 90% for recognizing single primitive gestures and slightly above 70% for composite gestures.
- Speed: The system operates at 15 frames per second on a 300 MHz Pentium Pro system with a Matrox Meteor frame grabber.
- Qualitative Findings: The system demonstrated viability for unstructured environments and robustness to different skin tones and lighting conditions. The authors observed that subjects improved their gesture accuracy when provided with visual feedback from the robot.
Limitations and Future Work
Limitations: * Shadows: The motion detection method can be affected by shadows of moving objects. * Tracking Failures: Tracking can fail when the hand leaves the camera's field of view. * Distortions: Slight distortions in diagonal motions can occur due to camera tilt. * User Errors: Subjects sometimes made errors in executing gestures, particularly diagonal or composite ones. * Slow Starts: Gestures were sometimes not detected if the user started the motion too slowly. * Composite Gesture Complexity: The geometric dependency and less natural execution of composite gestures contributed to their lower accuracy.
Future Work: * Improved Segmentation: Develop better methods for differentiating the hand from other moving body parts and continuously tracking the hand. * Simultaneous Motion: Address scenarios involving simultaneous motion of both the robot and the human. * Multi-Person Interaction: Support simultaneous interaction with multiple people. * 3D Gestures and Hand Poses: Extend the system to interpret hand poses and 3D gestures. * Gesture Learning: Implement gesture learning capabilities for the robot. * Robot Reaction and Feedback: Integrate the robot's reaction to perceived gestures and its own "feelings" into the interaction. * Interpreting Attitudes and Emotions: Incorporate the interpretation of human attitudes and emotions implicit in gestures.
Summary updated