Incorporating Gesture and Gaze into Multimodal Models of Human-to-Human Communication

Lei ChenNAACL 2006

TL;DR

This paper explores how to improve the understanding of human communication by integrating non-verbal cues like gestures and gaze into computational models. By combining these visual signals with speech and prosody, the authors aim to more accurately detect structural events in conversations, leading to better analysis of human interaction.

What Problem Does This Paper Address?

Understanding human communication is complex because it's not just about the words spoken. Spoken language is inherently structured (sentences, turns, topics), but these structures aren't always explicit in the text. To build better models of communication, we need to detect these underlying structures. While lexical and prosodic information (like tone of voice) have been used, human communication also heavily relies on non-verbal cues such as gestures and gaze. These visual signals are tightly linked in time and meaning to speech, suggesting they hold crucial information for understanding communication structure. The challenge is to effectively incorporate these multimodal cues into computational models to improve the detection of structural events like sentence boundaries, speech repairs, and conversational floor control.

Methods

The research presented is built upon a data-driven, multimodal approach. Key methods include:

  • Multimodal Corpus Collection: The authors have focused on building and utilizing multimodal corpora that capture synchronized audio, video, and motion tracking data. This includes a dialogue corpus and a meeting corpus designed for studying human interaction.
  • Feature Extraction: They extract features from various modalities:
  • Lexical: Standard word transcriptions.
  • Prosodic: Features derived from speech pitch and intensity.
  • Gestural: 3D hand position, "hold" states (lack of motion), and "effort" (hand movement kinetic energy).
  • Gaze and Body Posture: These are identified as important future additions.
  • Model Construction: They employ a multimodal model that integrates predictions from individual models for each knowledge source (lexical, prosodic, gestural). A common approach involves using CART decision trees for prosodic and gestural models and a hidden event language model for lexical information. These individual predictions are then combined, often using an HMM-based model combination scheme, to make a final prediction for structural events.

Datasets

  • Multimodal Dialogue Corpus: Collected under NSF KDI award, containing stereo video, transcriptions, prosodic analyses, and hand tracking data.
  • VACE Multimodal Meeting Corpus: Collected under sponsorship of ARDA VACE II, featuring synchronized audio, video, and motion tracking (head, torso, hand) from multi-party meetings. This corpus is used to study gestures, gaze, and floor control in more complex social interactions.

Key Insights and Ideas

  • Gestures and Speech are Tightly Linked: Studies suggest gesture and speech stem from a single underlying mental process, making gestures a valuable cue for understanding speech.
  • Non-Verbal Cues Complement Speech: Gestures, in particular, provide important additional information that can significantly improve the detection of structural events, complementing prosodic and lexical cues.
  • Speech Repairs and Gestures: Modification gestures (those that change state during a repair) are highly correlated with content modification speech repairs, offering a way to identify these specific types of errors.
  • Gaze and Floor Control: Gaze exchange and pointing gestures play a role in coordinating multi-party conversations and managing conversational floor control.
  • Multimodal Models Outperform Unimodal: Integrating multiple modalities (lexical, prosodic, gestural) leads to lower detection error rates for structural events compared to using any single modality alone.

Key Results

  • Improved Structural Event Detection: By incorporating gesture features, models achieve lower overall detection error rates for sentence units (SUs) compared to models relying solely on lexical and prosodic information.
  • Identification of Speech Repair Types: Gesture patterns, specifically modification gestures, can help distinguish between different types of speech repairs (e.g., content modification vs. repetition).
  • Insights into Floor Control: Analysis of multimodal cues (speech and visual) has identified patterns helpful for predicting floor control events in meetings, such as gaze exchange and discourse markers.

Limitations and Future Work

  • Expanding Non-Verbal Cues: The current work primarily focuses on gestures. Future research aims to expand the non-verbal features to include eye gaze and body posture.
  • Alternative Integration Architectures: The authors plan to investigate integration architectures beyond the HMM shown in Figure 2.
  • Understanding Mutual Impacts: A key future direction is to investigate the mutual impacts between different structural events, such as the relationship between sentence units (SUs) and floor control structure.
  • Summarizing Meetings: The detected structural events can be used to enhance meeting understanding, with a particular interest in locating salient portions of a meeting using multimodal cues for summarization.
  • Real-time Processing: While not explicitly stated as a limitation, the complexity of multimodal processing suggests that real-time application would be a significant future challenge.

Summary updated

BibTeX
@inproceedings{chen2006incorporating,
  author = {Lei Chen},
  title = {{Incorporating Gesture and Gaze into Multimodal Models of Human-to-Human Communication}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure