Placing Objects in Gesture Space: Toward Incremental Interpretation of Multimodal Spatial Descriptions

Ting Han, Casey Kennington, David SchlangenAAAI 2018 · DOI

Introduction

When humans give directions or describe spatial layouts, they frequently use "gesture space"—an imaginary coordinate system in the air where they "place" landmarks that are not physically present. For instance, a speaker might point to a spot in front of them to represent a bus stop and then point to the right to indicate a nearby hotel. For robotic systems to navigate effectively alongside humans, they must be capable of interpreting these abstract multimodal descriptions in real-time. This paper presents a system designed to incrementally fuse speech and deictic gestures to ground spatial descriptions into mental maps.

System Architecture

The proposed system employs a real-time pipeline that processes language and gestures in parallel before merging them in a fusion module.

The multimodal processing pipeline, illustrating the parallel tracks for language and gesture processing leading to a fusion module.

Gesture Processing

Using a Leap Motion sensor, the system tracks 92 raw hand features. An LSTM network classifies these frames to identify "stroke holds"—the meaningful phases of a pointing gesture. These 3D coordinates are then projected into a 2D coordinate system using Gaussian Kernel Density Estimation (KDE) to represent the speaker's intended spatial layout.

Language Processing

The language track uses an LSTM for utterance segmentation, determining when a description of one object ends and another begins. Natural Language Understanding (NLU) is handled by the Simple Incremental Update Model (SIUM), which maps specific words (e.g., "red," "square," "left") to object properties and spatial relations.

Multimodal Fusion

The system utilizes a "late fusion" approach, combining probability distributions from both speech and gesture tracks. A key innovation is the evaluation of spatial configurations; the system analyzes the relative angles and distances between a sequence of gestures to find the best match among candidate scenes.

Experimental Setup

To evaluate the system, the authors collected a Multimodal Spatial Description Corpus. Thirteen participants described 100 different scenes containing simple geometric shapes (circles and squares of various colors and sizes). This setup maintained spatial complexity while reducing linguistic ambiguity. The resulting dataset includes audio, video, and high-frequency hand-motion data, which has been made publicly available.

Results and Discussion

The system was tested on a scene-retrieval task, where it had to identify the correct scene from six candidates. The multimodal approach significantly outperformed unimodal baselines:

Condition Accuracy Mean Reciprocal Rank (MRR)
Speech Only 0.75 0.79
Gesture Only 0.50 0.50
Multimodal (Speech + Gesture) 0.84 0.85

Beyond final accuracy, the system's incremental performance is a critical metric. As shown in the figure below, the inclusion of gestures allows the system to reach a higher Mean Reciprocal Rank (MRR) earlier in the utterance compared to speech alone.

Incremental performance comparison showing that the multimodal system (speech + gesture) consistently achieves higher MRR than speech alone as the utterance unfolds.

The multimodal system reached its "first final" correct decision at 65% of the utterance duration, a 600ms improvement over the speech-only condition (67%). This suggests that gestures provide redundant but vital cues that help the system resolve ambiguity more quickly.

Conclusion

This research demonstrates that robots can effectively "see" landmarks in gesture space by fusing verbal cues with deictic gestures. While the current study focused on simplified 2D environments, the results indicate that multimodal fusion not only increases overall accuracy but also enhances the speed of incremental interpretation. Future work will explore more complex 3D environments and the integration of Automatic Speech Recognition to test the system's robustness in real-world, noisy conditions.

Summary updated

BibTeX
@inproceedings{han2018placing,
  author = {Ting Han and Casey Kennington and David Schlangen},
  title = {{Placing Objects in Gesture Space: Toward Incremental Interpretation of Multimodal Spatial Descriptions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11974},
}
Figure