Recognizing and Interpreting Gestures on a Mobile Robot

AAAI 1996

TL;DR

This paper presents a real-time, on-board 3D gesture recognition system for mobile robots that allows for natural human-robot interaction without special equipment or static environments. The system uses a novel "proximity space" method guided by a coarse human model to interpret gestures like pointing and halting, integrating them into an intelligent control architecture for complex task execution.

What Problem Does This Paper Address?

For mobile robots to effectively collaborate with humans, especially in dynamic environments like space or planetary surfaces, they need to understand natural human communication cues beyond just speech. Gestures are crucial for clarifying ambiguous commands and efficiently conveying geometric information. However, implementing gesture recognition on a mobile robot presents significant challenges: limited onboard processing power, the need to operate in moving and unpredictable environments, difficulties with precise camera calibration, and the requirement for real-time performance that matches human interaction speeds. This paper tackles these challenges by developing a robust, onboard gesture recognition system that can interpret gestures in context and drive robot actions.

Methods

The core of the system relies on a stereo vision system mounted on a pan/tilt/verge head. The novelty lies in the "proximity space" method and the use of a coarse 3D human model.

  • Proximity Spaces: Instead of processing the entire visual scene, the system focuses attention on small 3D regions called "proximity spaces." These spaces dynamically track visual texture and motion, acting like reactive agents that compete for control to maintain their position and size. This active vision approach limits computational load and focuses on relevant areas.
  • Coarse 3D Human Model: A simplified skeletal model of a human (head, shoulders, arms, hands) is used to constrain the movement of multiple linked proximity spaces. This model helps maintain tracking of body parts even in noisy conditions and provides the kinematic information needed to interpret gestures based on joint angles.
  • Gesture Definition: Gestures are defined by the relative angles between the links of the 3D model, specifically focusing on upper and lower arm angles. A confidence metric is built over time to report a gesture once a threshold is met, preventing spurious detections.
  • Integration with Intelligent Control: The gesture recognition system is integrated with an intelligent agent architecture (3T) that uses Reactive Action Packages (RAPs). This allows gestures to be interpreted within specific task contexts, enabling complex robot behaviors like moving to a pointed location, acquiring an object along a vector, or tracking an agent.

Datasets

Not discussed in the paper. The experiments described appear to be conducted with human subjects interacting with the mobile robot in real-time.

Key Insights and Ideas

  • Active Vision for Efficiency: Focusing computational resources on "proximity spaces" is a key insight for enabling real-time, onboard processing on a resource-constrained mobile robot. This is crucial for handling dynamic environments.
  • Model-Guided Tracking: Using a coarse 3D human model to constrain the movement of visual trackers (proximity spaces) is a robust way to maintain tracking of body parts, even with real-world noise and occlusions.
  • Context is King for Gestures: The paper strongly emphasizes that the meaning of a gesture (e.g., pointing) is highly dependent on the task context. Integrating gesture recognition with an intelligent control architecture (RAPs) is essential for making gestures truly useful for robot interaction.
  • Gesture Confidence for Robustness: Building confidence in a gesture over time, rather than immediate detection, significantly reduces false positives and leads to more reliable gesture recognition.

Key Results

  • Real-time Performance: The system operates in real-time, tracking gestures at frame rate (30 Hz) and identifying them at speeds natural to a human.
  • Range of Operation: Gestures can be recognized from distances as close as 1.25 meters up to 5 meters.
  • Tracking Speed: The system can track a fully extended arm moving at speeds up to approximately 36 degrees per second.
  • Pointing Accuracy: For pointing gestures at distances between 2.5 and 4.5 meters, the mean error distance was 0.41 meters (standard deviation of 0.17). Accuracy degraded at greater distances.
  • Height Estimation: The system can automatically estimate a person's height with an average error of 0.07 meters, which is sufficient for the gesture recognition model.
  • Task Integration: The system successfully demonstrated interpreting gestures within task contexts, such as using pointing to designate a target for the robot to move to or to acquire an object.

Limitations and Future Work

  • Single Arm Tracking: The current system can only track one arm at a time due to processing limitations and the field of view constraints when both arms are extended.
  • Limited Gesture Set: The system currently recognizes six distinct gestures and does not support gestures that occur over time (e.g., waving).
  • Distance Degradation: Pointing accuracy decreases significantly at longer distances.
  • Future Work: The authors plan to extend the system to recognize gestures over time and integrate gesture recognition with speech recognition for more comprehensive human-robot communication. They also aim to improve the robustness of the system and its ability to handle more complex tasks and environments.

Summary updated

BibTeX
@inproceedings{1996recognizing,
  title = {{Recognizing and Interpreting Gestures on a Mobile Robot}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1996},
}

Loading related content...

Figure