Influence of gestural salience on the interpretation of spoken requests

Gideon Kowadlo, Patrick Ye, Ingrid ZukermanInterspeech 2010

TL;DR

This paper introduces a probabilistic model that integrates pointing gestures with spoken language to improve the interpretation of user requests for robots. By considering the spatial and temporal salience of gestures, the system significantly enhances its ability to understand what the user intends, especially when speech recognition errors are present.

What Problem Does This Paper Address?

Interpreting human commands, especially for robots interacting in real-world environments, is a complex challenge. Users often combine spoken language with non-verbal cues like pointing gestures to refer to objects. A key difficulty lies in accurately linking these gestures to the intended objects, especially when the speech recognition system makes errors or when multiple objects are present. Existing systems often treat speech and gesture as separate inputs or use simpler fusion methods. This paper tackles the problem of how to effectively and probabilistically integrate pointing gestures into a spoken language understanding system to resolve ambiguity and improve accuracy.

Methods

The authors extend their existing spoken language interpretation module, "Scusi?", by incorporating a salience-based mechanism for pointing gestures. The core idea is that pointing makes objects more salient, influencing their probability of being the intended referent.

The approach models the influence of pointing gestures on the probability of an object being the intended referent, $Pr(k|C)$, by considering two components:

  • Spatial Salience ($P_s$): This is estimated using a conic Gaussian density function centered around a "pointing line" derived from the gesture's direction. The probability is influenced by the distance of an object to this line and whether it's occluded by other objects. A key innovation here is the use of a projected distance to account for pointing imprecision and the variance of the Gaussian increasing with this projected distance.
  • Temporal Salience ($P_t$): This is modeled by considering the timing of the pointing gesture relative to the timing of words in the utterance that refer to objects. A Gaussian time distribution is used to capture this temporal correlation.

These spatial and temporal probabilities are then combined using sigmoid functions to influence the overall probability of an object being the intended referent. The system's overall interpretation probability is a weighted combination of probabilities from speech recognition, parsing, and this new gesture integration module.

Datasets

The authors collected a new corpus for evaluation.

  • Corpus: A user study was conducted in a room with 33 labeled objects. Participants were asked to request 12 specific items. The corpus consists of 212 requests, with 105 involving pointing gestures.
  • Data Collection: To overcome limitations in real-time gesture recognition and ASR, pointing gestures were manually encoded from video recordings, and utterances were slightly sanitized and fed into the ASR system.

Key Insights and Ideas

  • Gestural Salience Matters: Pointing gestures significantly improve the interpretation accuracy of spoken requests, especially when the speech recognition system is imperfect.
  • Spatial and Temporal Integration: Effectively combining both the spatial location and the timing of a pointing gesture is crucial for accurate interpretation.
  • Handling Imprecision: The method for estimating spatial salience accounts for pointing imprecision by adjusting the probability distribution based on the projected distance of an object from the pointing line.
  • Occlusion Awareness: The model incorporates a factor to reduce the probability of an object being the target if it is occluded by other objects closer to the user.
  • Overcoming ASR Errors: The impact of pointing is more pronounced when using ASR input compared to text input, suggesting that gesture information can help disambiguate utterances that are already noisy due to speech recognition errors.

Key Results

The evaluation compared "Scusi?-Pointing" (with the gesture module activated) against "Scusi?-NoPointing" (without the gesture module).

  • Sub-corpus with Pointing Gestures:
    • For ASR input, the percentage of Gold ICGs in the top 1 interpretation increased from 74.3% (Scusi?-NoPointing) to 76.2% (Scusi?-Pointing).
  • The average adjusted rank (AR) significantly improved from 10.01 to 7.45 (p < 0.01).
  • For Text input, the AR improved from 3.31 to 1.90 (p < 0.01).
  • Sub-corpus without Pointing Gestures (Artificial Pointing):
    • Introducing pointing artificially led to smaller, but still statistically significant, improvements in AR (e.g., from 4.73 to 3.12 for ASR input, p < 0.05).
  • Overall: The results confirm that pointing information improves interpretation performance, particularly when dealing with imperfect ASR.

Limitations and Future Work

  • Simulated Pointing: The current evaluation relies on simulated pointing gestures due to limitations in real-time gesture recognition systems. The authors plan to study the sensitivity of their mechanism by incorporating actual pointing error.
  • ASR Limitations: The ASR system used has limitations (speaker-dependent, cannot recognize certain words), which required sanitizing utterances.
  • Gesture Recognition System Limitations: The gesture recognition system requires gestures to be held for 2 seconds.
  • Future Work: The authors aim to study the sensitivity of their mechanism to pointing errors and explore more complex utterances and gesture types.

Summary updated

BibTeX
@inproceedings{kowadlo2010influence,
  author = {Gideon Kowadlo and Patrick Ye and Ingrid Zukerman},
  title = {{Influence of gestural salience on the interpretation of spoken requests}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure