TL;DR
This paper introduces the KTH-Idiap Group-Interviewing corpus, a novel multimodal dataset designed to capture rich conversational dynamics in group settings. By combining high-quality audio, precise eye-gaze tracking, and depth data, this corpus enables deeper analysis of complex social behaviors like group involvement and individual engagement, which are crucial for advancing human-computer interaction.
What Problem Does This Paper Address?
Analyzing multi-party conversations is essential for understanding complex social interactions, from team dynamics to human-robot collaboration. While existing multimodal corpora have made strides, there's a notable gap in datasets that offer synchronized, high-fidelity data for group discussions, specifically including accurate gaze tracking for all participants. This limitation hinders research into higher-level constructs like group involvement, individual engagement, and rapport, which heavily rely on multimodal cues. The authors aim to fill this gap by creating a corpus that captures these nuances in a controlled yet dynamic group interview setting.
Methods
The core contribution is the creation and description of the KTH-Idiap Group-Interviewing corpus. The recording setup involved four participants (one interviewer, three interviewees) in a group interview scenario. Key methodological aspects include:
- Multimodal Sensing: The setup utilized close-talking mono-directional microphones for high-quality audio, Windows Kinect sensors for RGB and depth data (enabling head and facial behavior analysis), and GoPro cameras for high-resolution visual data.
- Synchronization: A robust synchronization mechanism was employed using a master audio signal and a visual LED cue. This ensured precise alignment across audio and video streams, crucial for analyzing subtle temporal dynamics.
- Calibration: A multi-stage calibration process was implemented, including Kinect stereo calibration, GoPro intrinsic and extrinsic calibration relative to Kinect, and finally, the 3D pose of Kinect sensors within a world coordinate system. This is vital for accurate spatial analysis of participant behavior.
- Corpus Design: The interview scenario was structured into five phases to elicit varied conversational dynamics, from spontaneous speech to collaborative project ideation. This design aimed to maximize variation in participant dynamics, engagement levels, and conversational flow.
Datasets
The paper introduces the KTH-Idiap Group-Interviewing corpus. This is a newly created benchmark dataset.
- Key Features: It comprises five multi-party, multi-modal group interviews, each lasting approximately one hour, totaling about 5 hours of recordings.
- Data Modalities: Includes synchronized audio (close-talking microphones), RGB video, depth data (from Kinects), and eye-gaze data.
- Annotations: The corpus includes manual gaze annotations for a subset of the data, voice activity, and short utterances (backchannels, hesitations, fillers). Additionally, participants completed the Big Five Questionnaire, and interviewers provided ratings on participant qualities.
- Distinction from Existing Corpora: Unlike corpora like AMI (scenario-driven, limited gaze), SONVB (dyadic, limited gaze targets), or D64/Wolf (limited gaze coverage), the KTH-Idiap corpus aims for comprehensive, high-quality gaze tracking across all participants in a dynamic group setting.
Key Insights and Ideas
- Intrinsic Motivation Matters: The corpus design leveraged intrinsic motivation (PhD students discussing their projects) to elicit more natural and engaged conversational behavior, which is crucial for studying genuine group dynamics.
- Gaze in Multi-Party Dynamics: Preliminary results suggest that gaze patterns observed in dyadic conversations (e.g., partner-oriented gaze during backchannels) extend to multi-party settings. The observer's gaze shifts towards the incoming speaker, potentially indicating anticipation or uncertainty about turn-taking.
- Observer Uncertainty: In speaker change events, the observer's increased gaze towards both the previous and incoming speakers hints at a potential strategy to gauge who will take the turn, a nuanced behavior not easily captured in simpler setups.
- Richness for HCI/HRI: The multimodal nature and structured dynamics make this corpus invaluable for developing more socially aware HCI/HRI systems that can better interpret and respond to group interactions.
Key Results
- The corpus successfully captured multimodal data (audio, RGB-D, eye-gaze) for five group interviews, totaling approximately 5 hours.
- Preliminary analysis of gaze patterns during "Overlap with Backchannel" events showed an increase in partner-oriented gaze for the previous speaker, similar to dyadic findings.
- The observer's gaze probability towards the incoming speaker increased before a backchannel, suggesting anticipatory behavior.
- During "Overlap with Speaker Change," the observer's gaze increased towards both the previous and incoming speakers, indicating potential uncertainty about turn-taking.
- The corpus allows for automatic retrieval of voice activity and eye-gaze annotations, reducing the need for expensive manual annotation.
Limitations and Future Work
- Limited Number of Interactions: The corpus contains only five group interactions, which may be insufficient for generalizing findings, especially concerning variables like extraversion. Further recordings are needed.
- Uncontrolled Variables: The corpus was not further controlled for the degree of acquaintance, culture, or gender. Future work could involve recordings with controlled variations in these factors to study their impact on group dynamics.
- Future Directions: The authors suggest using the corpus for investigating the effect of extraversion in group interviews, studying gaze patterns in relation to group involvement and individual engagement, and developing person-independent or person-specific gaze models for accurate visual focus of attention (VFOA) estimation. The rich data also supports multimodal analysis of turn-taking behaviors and the modeling of group involvement, individual engagement, and joined attention.
Summary updated