Prediction of turn-taking by combining prosodic and eye-gaze information in poster conversations

Tatsuya Kawahara, Takuma Iwatate, Katsuya TakanashiInterspeech 2012

TL;DR

This paper tackles the challenge of predicting turn-taking in poster sessions, a common but understudied multi-party conversation setting. By combining prosodic cues from the presenter with eye-gaze and backchannel information from the audience, the authors achieve promising results in predicting both when a turn will be yielded and who will take it. This work is crucial for developing more intelligent systems for automated recording and conversational agents in such interactive environments.

What Problem Does This Paper Address?

Turn-taking, the natural dance of conversation where participants alternate speaking turns, is complex, especially in multi-party settings like meetings or informal discussions. While much research has focused on dyadic (two-person) conversations, multi-party interactions present a greater challenge: predicting not just if a turn will be yielded, but to whom. Poster sessions, common at academic conferences, are a particularly interesting case. Here, the presenter typically dominates the speaking turns, making audience contributions more informative and thus more critical to capture. Existing models for turn-taking often overlook the unique dynamics of poster sessions, where participants interact with both each other and the presented material (the poster). This paper aims to fill this gap by developing a predictive model for audience turn-taking in these specific conversational contexts.

Methods

The authors employ a machine learning approach, specifically using Support Vector Machines (SVM) and logistic regression, to predict turn-taking events. They extract a rich set of features from multi-modal data:

  • Prosodic Features: Standard features like pitch (F0) and power (mean, max, min, range) of the presenter's speech are used as a baseline.
  • Eye-Gaze Features: This is a key innovation. They analyze not only what participants are looking at (poster, presenter, or other audience members) but also joint eye-gaze events (e.g., presenter and audience looking at each other simultaneously). They also consider the duration of these gaze events.
  • Backchannel Features: They capture both verbal (e.g., "hai") and non-verbal (e.g., nodding) cues from the audience, hypothesizing that these indicate engagement and potential intent to speak.

The prediction task is split into two sub-tasks:

  1. Speaker Change Prediction: Predicting if the presenter will yield the turn to the audience.
  2. Next Speaker Prediction: If a turn is yielded, predicting which audience member will speak.

Datasets

The study utilizes a custom dataset recorded from four Japanese poster sessions. Each session involved one presenter and two audience members. The data includes segmented speech (IPUs), speaker labels, transcriptions, and manually annotated verbal backchannels. Crucially, eye-gaze information was captured using eye-tracking recorders and motion capture systems, and gaze targets were identified by matching gaze vectors to participant and poster positions. This multi-modal dataset, specifically designed for poster session interactions, is a valuable contribution.

Key Insights and Ideas

  • Eye-Gaze is Crucial for Turn-Taking: The presenter's gaze towards an audience member just before yielding the turn is a strong indicator of who will speak next. This highlights the importance of non-verbal cues in signaling conversational intent.
  • Joint Gaze Matters: Mutual gaze between the presenter and an audience member ("li" event) is more indicative of turn-taking than simply looking at the poster.
  • Backchannels Signal Intent: Audience members who make more backchannels (both verbal and non-verbal) are more likely to take the turn. This suggests these are not just passive acknowledgments but active signals of engagement.
  • Combining Modalities is Key: While individual features offer some predictive power, combining prosodic and eye-gaze information significantly improves the prediction of speaker change. Similarly, combining eye-gaze and backchannel features boosts the accuracy of predicting the next speaker.
  • Poster Session Dynamics are Unique: The unbalanced nature of turns (presenter dominates) and the interaction with the poster itself necessitate specific modeling approaches, differentiating it from general multi-party conversation studies.

Key Results

  • Speaker Change Prediction: Combining prosodic features of the presenter with eye-gaze features of all participants yielded the best results, achieving a recall of 0.706 and an F-measure of 0.319. While the F-measure might seem modest, it's important to note that audience turn-taking constitutes only 11.9% of all turns, making this a challenging "detection" problem rather than a balanced classification task.
  • Next Speaker Prediction: The combination of all features (eye-gaze object, joint eye-gaze events, duration, and backchannels) achieved the highest accuracy at nearly 70%. Simple eye-gaze features alone, when incorporating duration, significantly improved accuracy over chance (50%).

Limitations and Future Work

  • Limited Dataset Size: The study was conducted on only four poster sessions. While cross-validation was used, a larger and more diverse dataset would strengthen the findings.
  • Focus on Two Audience Members: The current setup involved only two audience members. Future work could explore scenarios with more participants.
  • Feature Engineering: While effective, the authors acknowledge that features could be extended to account for multiple audience members simultaneously.
  • Future Directions: The authors plan to use these findings to design a "smart poster-board" capable of controlling recording equipment and annotating audience reactions. This technology could also power more intelligent conversational agents for autonomous presentations.

Summary updated

BibTeX
@inproceedings{kawahara2012prediction,
  author = {Tatsuya Kawahara and Takuma Iwatate and Katsuya Takanashi},
  title = {{Prediction of turn-taking by combining prosodic and eye-gaze information in poster conversations}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure