Abstract
As robots become more ubiquitous, it is increasingly important for untrained users to be able to interact with them intuitively. In this work, we investigate how people refer to objects in the world during relatively unstructured communication with robots. We collect a corpus of deictic interactions from users describing objects, which we use to train language and gesture models that allow our robot to determine what objects are being indicated. We introduce a temporal extension to state-of-the-art hierarchical matching pursuit features to support gesture understanding, and demonstrate that combining multiple communication modalities more effectively capture user intent than relying on a single type of input. Finally, we present initial interactions with a robot that uses the learned models to follow commands.
Introduction
The integration of robots into everyday environments necessitates interaction methods that are intuitive for non-expert users. Traditional human-robot interaction (HRI) systems often rely on rigid command lexicons or predefined gesture sets, imposing a significant cognitive burden on the user. This research addresses the problem of grounded language acquisition: enabling a robot to autonomously associate natural, unscripted speech and physical gestures with objects in its environment.
The focus is on "directing-to" interactions, specifically deictic gestures. Unlike controlled laboratory settings, natural human pointing is highly variable, involving multi-finger points, circular motions, and broad sweeps, often accompanied by ungrammatical or noisy language.
Methodology
The proposed system utilizes RGB-D data from a Kinect sensor to process interactions through a multi-stage pipeline.
1. Perception and Segmentation
The system segments tabletop objects using RANSAC plane fitting. Hands are identified as dynamic point clusters extending into the workspace.
2. Temporal Gesture Classification
A key contribution is the temporal extension of Hierarchical Matching Pursuit (HMP). While HMP is typically applied to static imagery, this work adapts it to pool features over time. A binary encoding in the K-SVD algorithm is employed to ensure the model preserves physical magnitude—specifically the distance between the hand and objects—which is critical for disambiguating intent.
3. Grounded Language and Visual Learning
Visual attributes such as color and shape are learned via HMP features extracted from object image patches. The language model uses a bag-of-words approach to statistically learn correspondences between spoken words and these visual attributes (e.g., associating "cuboid" with a cube visual classifier) without manual labeling.
4. Multimodal Fusion
The system integrates probabilities from gesture, language, and visual models. This additive fusion ensures robustness; for instance, if a user's speech is muffled, the gesture model can still provide strong evidence for the target object.
Experimental Results
The system was evaluated using a new RGB-D corpus of 364 annotated videos featuring 13 participants in 234 tabletop scenes.
Performance Metrics
The integrated multimodal system achieved a peak F1 score of 80.7%. The temporal HMP gesture model significantly outperformed baseline "closest-approach" heuristics, reaching 91% precision at 80% recall compared to the baseline's ~63% precision.
Real-World Validation
In live trials with five novel participants, the robot demonstrated its ability to generalize, achieving 79% precision and 90% recall in identifying and manipulating objects despite speech recognition errors and varied user behaviors.
Discussion and Key Insights
- Dynamic Gestures: Natural pointing is a time-series event. Capturing "dynamic signatures" like circular loops or sweeps is essential for accurate intent recognition.
- Magnitude Preservation: Maintaining scale information in sparse coding is vital for HRI, as the proximity of a hand to an object is a primary indicator of reference.
- Complementary Modalities: Language typically specifies "what" an object is, while gesture specifies "where" it is. Their combination is significantly more powerful than either modality in isolation.
Limitations and Future Work
A primary bottleneck remains speech recognition accuracy, with 46% of language-related errors attributed to the underlying API. Additionally, users frequently gestured outside the sensor's field of view. Future research will explore more sophisticated natural language processing, such as CCG parsers, and expand the system's vocabulary to include a broader range of complex real-world objects.
Summary updated