TL;DR
This paper explores how eye-gaze and speech cues work together to manage turn-taking in conversations. They found that eye-gaze is a surprisingly strong predictor of turn management, even more so than speech features when the speaker's intent is clear, suggesting a crucial role for shared attention in smooth dialogue flow.
What Problem Does This Paper Address?
Fluent human conversation relies on intricate coordination of who speaks when. This "turn-taking" is managed through a complex interplay of verbal and non-verbal cues. While speech timing and content are obvious factors, the role of non-verbal signals like eye-gaze in this coordination is less understood, especially in computational models. This paper investigates how eye-gaze, alongside speech prosody and dialogue acts, contributes to predicting and managing turn-taking events in natural dialogues. The goal is to build more effective conversational agents that can better understand and participate in human-like interactions.
Methods
The researchers employed a machine learning approach, specifically Support Vector Machines (SVMs), to classify turn-taking events. They extracted three types of features from recorded conversations:
- Dialogue Acts: These represent the communicative function of an utterance (e.g., asking a question, providing information, backchanneling).
- Speech Prosody: Features like pitch (F0) and loudness (power) were extracted, along with their maximum, minimum, mean, and range values. These were normalized per speaker to account for individual differences.
- Eye-Gaze: This included information about where the speaker was looking (e.g., at an interlocutor, an object, or elsewhere).
These features were combined into a single vector representation for each utterance. Classification experiments were conducted using different combinations of these features to assess their individual and combined impact on predicting turn-give and turn-hold events.
Datasets
The study utilized a custom dataset of natural, free-flowing three-party dialogues. The data was collected from six speakers in a triangular formation, with one speaker's eye-gaze being tracked using an eye-tracker, while the other two were video-recorded. The dialogues were annotated for dialogue acts, gaze, face, head movements, turn-taking, feedback, and emotion/attitude by three annotators. The inter-annotator agreement was measured at a kappa value of 0.46, indicating moderate agreement. The analysis focused on 4-6 minute excerpts from four different dialogues, resulting in 110 segmented utterance units.
Key Insights and Ideas
- Eye-gaze is a powerful turn-taking signal: The results strongly suggest that eye-gaze is not just a passive indicator of attention but an active signal in turn management.
- Eye-gaze can be more important than speech: When the intention of an utterance is clear (as indicated by dialogue acts), eye-gaze features were found to be even more important than speech features for predicting turn-taking. This highlights the role of shared attention in smooth dialogue.
- Parallel, not complementary, effects: Eye-gaze and speech features seem to contribute to turn management in parallel rather than complementing each other significantly. This implies they offer distinct but equally valuable information.
- Turn-hold prediction benefits greatly from gaze: The accuracy of distinguishing between a speaker holding the turn and preparing to yield it (turn-hold) significantly improved when eye-gaze features were included.
Key Results
The classification experiments showed that:
- Using only dialogue act features resulted in an F-measure of 0.581.
- Adding speech features to dialogue acts did not improve accuracy (F-measure remained 0.581).
- Adding eye-gaze features to dialogue acts significantly improved accuracy to an F-measure of 0.753.
- Including all three feature sets (dialogue act, speech, and eye-gaze) yielded the same F-measure of 0.753, indicating that speech features did not add further predictive power when eye-gaze was already present.
- Confusion matrices revealed that distinguishing between turn-give and turn-hold was particularly challenging with dialogue act and speech features alone, but eye-gaze greatly improved this distinction.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Limited scope of features: While eye-gaze was crucial, the study primarily focused on gaze direction. Future work should explore other aspects of eye-gaze, such as the entire eye region (eyebrows, eyelids, wrinkles), and how they correlate with turn-taking.
- Broader non-verbal cues: The paper suggests incorporating other non-verbal signals like head turns, upper body movements, and gestures, which were noted as important in previous work, especially in multi-party dialogues.
- Interaction dynamics: The study could be extended to investigate how long and how often a partner is looked at, as these temporal aspects of gaze are likely relevant to communicative intent.
- Individual differences: While speech features were normalized, further exploration of how individual speaker characteristics influence the interplay of gaze and speech in turn-taking could be beneficial.
- Complementary vs. Parallel contributions: While the current study found parallel contributions, further research might uncover more nuanced complementary roles under different conversational conditions.
Summary updated