Analysis of gaze and speech patterns in three-party quiz game interaction

Samer Al Moubayed, Jens Edlund, Joakim GustafsonInterspeech 2013

TL;DR

This paper introduces a novel multimodal dataset for studying three-party interactions, combining synchronized gaze and speech data from participants engaged in a quiz game. This rich dataset, collected using automated, non-intrusive methods, aims to bridge the gap in existing resources and enable deeper analysis of conversational dynamics, particularly engagement cues.

What Problem Does This Paper Address?

Understanding and modeling human-to-human interaction, especially in multiparty settings, is crucial for developing more natural and effective artificial conversational partners. While previous research has explored two-party interactions and multimodal cues like speech and head movements, there's a significant lack of large-scale, objective data that captures simultaneous gaze and speech patterns across multiple participants in spontaneous, engaging scenarios. Analyzing gaze behavior in multiparty settings is particularly challenging and time-consuming with traditional methods. This paper addresses this gap by presenting a method and a corpus for collecting and analyzing such data, focusing on engagement as a key phenomenon.

Methods

The authors developed a sophisticated data collection setup for three-party interactions. Participants engaged in a high-stakes quiz game, seated around a round table.

  • Gaze Tracking: Three state-of-the-art Tobii X2 gaze trackers were used, one per participant, positioned to capture eye movements. To address the limitation of connecting multiple trackers to a single computer, each tracker was connected to a separate machine.
  • Audio Recording: A high-quality Microcone™ microphone array, featuring six directional microphones and one omnidirectional stereo microphone, was employed. This array captures directed and de-noised audio, along with microphone activity and loudness estimates, enabling automatic Voice Activity Detection (VAD).
  • Video Recording: High-resolution USB3 video cameras were used to record each participant's face, capturing the same field of view as the gaze trackers. This allows for future analysis of facial movements.
  • Synchronization: A custom Java-based environment utilizing the SDKs of the equipment was developed to ensure automatic synchronization and alignment of gaze, audio, and video data across all recording machines. Post-recording synchronization was also performed using audio streams and timestamps.
  • Gaze Data Processing: Raw gaze data was clustered into recognizable gaze targets (e.g., on-head gaze, off-head gaze) using a 4x2-dimensional Gaussian mixture model and the Expectation Maximization (EM) method. This automated clustering reduces manual annotation effort.

Datasets

The paper introduces the Three-party Estimation Quiz Game corpus. This corpus consists of five one-hour recordings of three-party quiz game interactions, totaling approximately five hours of data. The interactions were highly engaged, with frequent speaker changes and minimal silence. This dataset is novel in its combination of synchronized, multi-participant gaze and speech data collected using automated techniques.

Key Insights and Ideas

  • Automated, Non-Intrusive Data Collection: The primary innovation is the development of a setup that allows for the automated and non-intrusive collection of rich multimodal data (gaze, speech, video) from multiple participants simultaneously. This significantly reduces the burden of manual annotation.
  • Gaze as a Key Engagement Cue: The paper highlights the importance of gaze patterns as a salient cue for conversational engagement, a critical aspect of human-machine interaction.
  • Challenges of Multiparty Gaze Analysis: The work acknowledges the inherent difficulties in analyzing gaze in multiparty settings and proposes automated clustering as a solution to overcome manual annotation limitations.
  • Synchronized Multimodal Analysis: The synchronized nature of the data allows for direct investigation of patterns across gaze and speech, enabling analysis of when participants are speaking, listening, or when all are silent.

Key Results

  • The collected corpus comprises five hours of highly engaged three-party conversations with a large number of speaker changes and very little silence.
  • Automatic VAD identified a total of 105 minutes of participant speech and 117 minutes of Game Master speech across the corpus, with only 30 minutes of silence.
  • Initial analysis revealed that speakers, on average, looked at either of the two listeners 57% of the time, with equal distribution between interlocutors.
  • When listening, participants looked at the speaker 47% of the time, at the co-listener 17% of the time, and at nobody for the remaining 36% of the time.
  • A significant proportion of "off-head-gaze" (where all three trackers detected gaze directed away from interlocutors) occurred when the Game Master was speaking or during complete silence.

Limitations and Future Work

  • Limitations: The paper notes that the Tobii gaze tracker has limitations regarding head movement boxes, necessitating participants to remain relatively still. The reliance on a single computer per tracker, while necessary, required careful synchronization.
  • Future Work: The authors plan to further investigate temporal speech-gaze patterns, exploring their dynamics in relation to overlaps, speaker changes, and third-party observer behavior. They also intend to analyze how these patterns vary across different states of the quiz game. Future work will also incorporate facial analysis (head rotation, facial expressions) using automated software.

Summary updated

BibTeX
@inproceedings{moubayed2013analysis,
  author = {Samer Al Moubayed and Jens Edlund and Joakim Gustafson},
  title = {{Analysis of gaze and speech patterns in three-party quiz game interaction}},
  booktitle = {Interspeech},
  year = {2013},
}
Figure