TL;DR
This paper investigates how listeners signal attention in face-to-face conversations using both vocal and visual cues (backchannels or BCs). They found that unlike in phone calls, rising/falling pitch isn't a strong BC cue, but gaze is. Importantly, the type of BC (vocal vs. visual) is influenced by the speaker's cues, suggesting a richer, multimodal approach is needed for conversational AI.
What Problem Does This Paper Address?
Conversational agents (like chatbots or virtual assistants) often struggle to exhibit natural human-like listening behavior. A key part of this is the use of "backchannels" (BCs) – short vocalizations ("uh-huh," "hmm") or visual cues (head nods) that signal attention, interest, and understanding without interrupting the speaker. Previous research on predicting BCs primarily focused on telephone conversations, relying heavily on vocal cues like pitch and pauses. However, in face-to-face interactions, the visual channel (gaze, facial expressions) becomes available, potentially altering the dynamics of BCs and the cues that invite them. This paper aims to bridge this gap by analyzing how vocal and visual cues from the speaker influence the timing and modality of listener backchannels in spontaneous, face-to-face dialogues.
Methods
The researchers analyzed an existing audiovisual corpus of spontaneous Dutch dialogues. They focused on identifying vocal backchannels (like "hmm") and visual backchannels (specifically head nods). Their analysis involved:
- Feature Extraction: They extracted prosodic features (pitch, energy) from the speaker's speech and gaze data from both the speaker and the listener (the "BC-er").
- Annotation: They manually annotated head nods and shakes, and used existing annotations for vocal backchannels. They also defined "bimodal BCs" as vocal and visual BCs occurring within a 0.2-second window.
- Statistical Analysis: They compared the frequency and timing of BCs in relation to various speaker cues, looking for significant differences between vocal and visual BCs. They used statistical tests (chi-squared) to determine if observed frequencies were significantly different from chance.
Datasets
- IFADV corpus: A publicly available audiovisual Dutch corpus containing spontaneous, unscripted face-to-face dialogues between acquainted participants. The corpus consists of 20 dialogs, each 15 minutes long. One dialog was excluded due to participants being overly aware of the recording.
Key Insights and Ideas
- Gaze is King (in Face-to-Face): Unlike telephone conversations where pitch contours are key, in face-to-face settings, gaze (especially mutual gaze between speaker and listener) is a strong predictor of backchannels. This makes intuitive sense – if you're looking at someone, they're more likely to see your signal of attention.
- Modality Matters: The type of backchannel (vocal vs. visual) is not random. Visual BCs are more likely to occur during the speaker's ongoing speech, while vocal BCs tend to happen during pauses. This suggests a strategic use of different channels based on the conversational flow.
- Pitch Cues Less Important: Rising or falling pitch, a significant cue in telephone BC prediction, was not found to be a strong BC-inviter in this face-to-face dataset. The availability of visual cues seems to reduce the reliance on vocal pitch for signaling attention.
- Energy as a Visual Cue Indicator: High energy in the speaker's speech was more strongly associated with visual BCs, while decreasing energy (often preceding pauses) was linked to vocal BCs. This hints that emphasized speech might invite a visual nod, while winding down speech might invite a vocal "uh-huh."
Key Results
- Vocal BCs in Pauses: Vocal BCs occurred in pauses 37% of the time, significantly higher than the chance of 28.4%.
- Visual BCs During Speech: Visual BCs were more likely to occur during the interlocutor's speech segment compared to vocal BCs.
- Gaze and BCs: Mutual gaze was significantly higher around BCs (57.3% of the time overall). Visual and bimodal BCs showed a stronger association with mutual gaze compared to vocal BCs.
- Pitch Cues: Low pitch regions occurred more often before BCs (26.5% vs. chance 18.1%), but there were no significant differences between vocal and visual BCs. Rising/falling pitch contours were not strong BC-inviting cues.
- Energy Cues: Visual BCs were significantly more likely to be preceded by high energy profiles (18.1% vs. 9.7% for vocal BCs).
Limitations and Future Work
- Focus on "Continuer" BCs: The study specifically excluded BCs with attitudinal or affective functions (like smiles or frowns), focusing only on those that signal the speaker should continue. Future work will explore these other types of BCs.
- Automatic Feature Extraction: While the study used both manual and automatic annotations, the authors acknowledge that real-time processing of speech and visual features for online BC prediction remains a significant challenge due to time constraints.
- Interchangeability of BCs: The paper suggests investigating whether vocal and visual BCs are truly interchangeable in certain contexts, which would require human perception studies rather than corpus analysis.
- Contextual Nuances: The authors plan to further analyze vocal and visual BC behavior in spontaneous dialogues, considering more nuanced contextual factors. They also aim to develop rule-based or machine learning models for automatic online prediction of BCs.
Summary updated