Contrasting cues to verbal and non-verbal backchannels in multi-lingual dyadic rapport

Gina-Anne Levow, Susan DuncanInterspeech 2012

TL;DR

This paper investigates how subtle vocal and non-verbal cues signal rapport in conversations across different cultures. By analyzing multi-lingual interactions, the researchers found that while both vocal backchannels and head nods serve to maintain engagement, they are often triggered by distinct prosodic cues, suggesting a more nuanced understanding of social resonance signals is needed.

What Problem Does This Paper Address?

Face-to-face interactions rely heavily on subtle cues to build and maintain rapport, which is crucial for effective communication, knowledge sharing, and learning. While previous research has identified various behaviors like nodding, gaze, and vocal backchannels that contribute to rapport, these cues are often culture-specific and their underlying mechanisms are not fully understood. Specifically, this paper addresses the gap in understanding how different types of social resonance signals, like vocal backchannels and head nods, are cued by the speaker's prosody, and how these patterns vary across different language and cultural groups. The authors highlight that most empirical studies focus on a single cue, and that backchannels are often treated as a monolithic class, overlooking potential functional differences.

Methods

The researchers employed a multi-modal, multi-lingual corpus of quasi-monologic story-telling interactions from dyads with established rapport. The core of their method involved analyzing speaker-side utterances that immediately preceded specific listener behaviors: vocal backchannels (BC), speaking turns (T), and head nods (HN), compared to instances with no listener verbal activity (N).

Key techniques included:

  • Feature Extraction: Prosodic features such as pitch (mean and slope), intensity (mean), voice quality (NHR), speaking rate, and utterance duration were extracted from speaker utterances. These features were computed over specific time windows preceding the listener's behavior.
  • Statistical Analysis: Kruskal-Wallis tests followed by Tukey post-hoc tests were used to identify statistically significant differences in prosodic features across the different listener behavior conditions (BC, T, N, HN). Significance was set at p < 0.05.
  • Integrated Analysis: The study moved beyond analyzing vocal backchannels and head nods in isolation by conducting a joint analysis, particularly focusing on instances where verbal backchannels and head nods occurred concurrently (HN-BC). This allowed for a comparison of cues preceding pure head nods (HN) versus those preceding joint head nods and verbal backchannels (HN-BC).

Datasets

The study utilized a previously collected multi-modal dyadic corpus. This corpus consists of:

  • Language/Cultural Groups: American English, Mexican Spanish, and Iraqi Arabic speakers.
  • Interaction Type: Quasi-monologic story-telling based on the "Pear Film" [23].
  • Dyad Composition: Close acquaintances or family members with assumed well-established rapport.
  • Data Modalities: Audio and video recordings, fully transcribed, time-aligned, and annotated for vocal backchannels, head nods, gestures, gaze, and blinks.
  • Subset Used: The paper specifically uses a subset of the corpus annotated for vocal backchannels and head nods, comprising approximately 13 Arabic (≈ 1 hour), 15 English (≈ 1.5 hours), and 15 Spanish (≈ 1 hour) dyads.

Key Insights and Ideas

  • Divergence in Cues: The most striking insight is the significant difference in prosodic cues that precede vocal backchannels versus head nods. While vocal backchannels are often cued by lower pitch and intensity in the preceding speaker utterance across languages, head nods are more variably cued, sometimes showing no significant prosodic difference from no-listener-activity contexts, and in some cases, being associated with higher pitch.
  • Head Nods are Not Monolithic: The analysis of concurrent head nods and verbal backchannels revealed that head nods occurring without a verbal backchannel (HN) are significantly longer than those that co-occur with a verbal backchannel (HN-BC). This suggests that head nods might serve different functions and are potentially cued by different speaker behaviors, with longer nods perhaps indicating a more sustained or different type of engagement.
  • Cross-Cultural Similarities and Differences: While there are language-specific patterns (e.g., pitch playing a more prominent role in Arabic for verbal backchannels), the general trend of lower pitch/intensity preceding vocal backchannels is observed across the studied groups, suggesting some universal aspects of signaling continued attention.
  • Implications for Rapport: The findings underscore that understanding social resonance signals requires a fine-grained, multi-modal, and cross-cultural approach. Treating all backchannels as a single class risks overlooking crucial nuances in how rapport is built and maintained.

Key Results

  • Vocal Backchannels: Across all three language groups, speaker utterances preceding vocal backchannels (BC) were associated with significantly lower pitch mean compared to contexts with no listener verbal activity (N). Lower intensity was also observed for BC in English and Spanish compared to N.
  • Head Nods: In contrast to vocal backchannels, head nods (HN) showed fewer consistent prosodic differences from no-listener-activity contexts (N). English and Spanish showed higher pitch for HN compared to N, while Arabic showed no significant prosodic differences for head nods.
  • Joint Analysis: Head nod durations were significantly longer for isolated head nods (HN) compared to those co-occurring with verbal backchannels (HN-BC) across all language groups. This supports the idea that these are distinct types of signals.
  • Table 2: Shows that vocal backchannels and turns occur at different rates across languages, with Arabic having the highest proportion of IPUs followed by listener turns or backchannels.
  • Table 4: Indicates the rates of head nods, with Arabic having the highest rate.

Limitations and Future Work

  • Limitations:
    • The study acknowledges the difficulty of POS tagging for dialectal Arabic and conversational Spanish, leading to the exclusion of POS tag sequence analysis.
  • Manual ToBI intonation annotation was not feasible due to resource constraints.
  • The corpus is based on quasi-monologic story-telling, which might not fully represent all types of dyadic interactions.
  • The "sentence completion" class was grouped with "vocal backchannel" due to its low rate, potentially masking specific cues for sentence completion.
  • Future Work:
    • The authors plan to further investigate how these prosodic cue contrasts reflect differences in the function or form of social resonance behaviors.
  • They aim to exploit these insights to improve automatic prediction of backchannels.
  • Further research is suggested to explore the functional differences between isolated head nods and those co-occurring with verbal backchannels.

Summary updated

BibTeX
@inproceedings{levow2012contrasting,
  author = {Gina-Anne Levow and Susan Duncan},
  title = {{Contrasting cues to verbal and non-verbal backchannels in multi-lingual dyadic rapport}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure