Learning from Unscripted Deictic Gesture and Language for Human-Robot Interactions

Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, Dieter FoxAAAI 2014 · OpenReview

Abstract

As robots become more ubiquitous, it is increasingly important for untrained users to be able to interact with them intuitively. In this work, we investigate how people refer to objects in the world during relatively unstructured communication with robots. We collect a corpus of deictic interactions from users describing objects, which we use to train language and gesture models that allow our robot to determine what objects are being indicated. We introduce a temporal extension to state-of-the-art hierarchical matching pursuit features to support gesture understanding, and demonstrate that combining multiple communication modalities more effectively capture user intent than relying on a single type of input. Finally, we present initial interactions with a robot that uses the learned models to follow commands.

Introduction

The integration of robots into everyday environments necessitates interaction methods that are intuitive for non-expert users. Traditional human-robot interaction (HRI) systems often rely on rigid command lexicons or predefined gesture sets, imposing a significant cognitive burden on the user. This research addresses the problem of grounded language acquisition: enabling a robot to autonomously associate natural, unscripted speech and physical gestures with objects in its environment.

Examples of unscripted deictic interactions where users describe objects using natural language and gestures.

The focus is on "directing-to" interactions, specifically deictic gestures. Unlike controlled laboratory settings, natural human pointing is highly variable, involving multi-finger points, circular motions, and broad sweeps, often accompanied by ungrammatical or noisy language.

Methodology

The proposed system utilizes RGB-D data from a Kinect sensor to process interactions through a multi-stage pipeline.

1. Perception and Segmentation

The system segments tabletop objects using RANSAC plane fitting. Hands are identified as dynamic point clusters extending into the workspace.

2. Temporal Gesture Classification

A key contribution is the temporal extension of Hierarchical Matching Pursuit (HMP). While HMP is typically applied to static imagery, this work adapts it to pool features over time. A binary encoding in the K-SVD algorithm is employed to ensure the model preserves physical magnitude—specifically the distance between the hand and objects—which is critical for disambiguating intent.

3. Grounded Language and Visual Learning

Visual attributes such as color and shape are learned via HMP features extracted from object image patches. The language model uses a bag-of-words approach to statistically learn correspondences between spoken words and these visual attributes (e.g., associating "cuboid" with a cube visual classifier) without manual labeling.

4. Multimodal Fusion

The system integrates probabilities from gesture, language, and visual models. This additive fusion ensures robustness; for instance, if a user's speech is muffled, the gesture model can still provide strong evidence for the target object.

Experimental Results

The system was evaluated using a new RGB-D corpus of 364 annotated videos featuring 13 participants in 234 tabletop scenes.

The Gambit robot platform executing a pick-and-place task based on multimodal user input.

Performance Metrics

The integrated multimodal system achieved a peak F1 score of 80.7%. The temporal HMP gesture model significantly outperformed baseline "closest-approach" heuristics, reaching 91% precision at 80% recall compared to the baseline's ~63% precision.

Real-World Validation

In live trials with five novel participants, the robot demonstrated its ability to generalize, achieving 79% precision and 90% recall in identifying and manipulating objects despite speech recognition errors and varied user behaviors.

Discussion and Key Insights

  • Dynamic Gestures: Natural pointing is a time-series event. Capturing "dynamic signatures" like circular loops or sweeps is essential for accurate intent recognition.
  • Magnitude Preservation: Maintaining scale information in sparse coding is vital for HRI, as the proximity of a hand to an object is a primary indicator of reference.
  • Complementary Modalities: Language typically specifies "what" an object is, while gesture specifies "where" it is. Their combination is significantly more powerful than either modality in isolation.

Limitations and Future Work

A primary bottleneck remains speech recognition accuracy, with 46% of language-related errors attributed to the underlying API. Additionally, users frequently gestured outside the sensor's field of view. Future research will explore more sophisticated natural language processing, such as CCG parsers, and expand the system's vocabulary to include a broader range of complex real-world objects.

Summary updated

Abstract

As robots become more ubiquitous, it is increasingly important for untrained users to be able to interact with them intuitively. In this work, we investigate how people refer to objects in the world during relatively unstructured communication with robots. We collect a corpus of deictic interactions from users describing objects, which we use to train language and gesture models that allow our robot to determine what objects are being indicated. We introduce a temporal extension to state-of-the-art hierarchical matching pursuit features to support gesture understanding, and demonstrate that combining multiple communication modalities more effectively capture user intent than relying on a single type of input. Finally, we present initial interactions with a robot that uses the learned models to follow commands.
BibTeX
@inproceedings{matuszek2014learning,
  author = {Cynthia Matuszek and Liefeng Bo and Luke Zettlemoyer and Dieter Fox},
  title = {{Learning from Unscripted Deictic Gesture and Language for Human-Robot Interactions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  url = {https://openreview.net/forum?id=HJ-jjag_-H},
}
Figure