Acoustic and Prosodic Correlates of Social Behavior

Agustín Gravano, Rivka Levitan, Laura Willson, Stefan Benus, Julia Hirschberg, Ani NenkovaInterspeech 2011 · DOI

Introduction

Human social interaction is mediated by subtle acoustic and lexical signals that inform our perceptions of others. While speech technology has made significant strides in detecting primary emotions like anger or joy, identifying nuanced social behaviors—such as likeability, the effort to be liked, and speech planning—remains a complex challenge. This research investigates how these social signals manifest in spontaneous speech and how they are interpreted by third-party observers.

Methodology

The study utilized the Columbia Games Corpus, specifically focusing on "Objects games" where a "Describer" guides a "Follower" in a task-oriented environment. The researchers employed a multi-stage analysis pipeline:

  1. Crowdsourced Annotation: 94 annotators on Amazon Mechanical Turk rated clips for social behaviors (e.g., "Does this person seem to be trying to be liked?").
  2. Acoustic Extraction: Features such as pitch, intensity, jitter, shimmer, and speaking rate were extracted using Praat software.
  3. Prosodic and Lexical Analysis: The ToBI (Tones and Break Indices) system was used for prosodic labeling, while disfluencies and emotional word strength (via Whissell’s Dictionary of Affect) were tracked.
  4. Statistical Modeling: Analysis was conducted at the Intonational Phrase (IP) level, using ANOVA and Kruskal-Wallis tests to identify significant correlates across gender dyads.

Social Cues and Gender Dynamics

A primary finding of the study is that social cues are highly contextual and performative, particularly regarding gender. The acoustic manifestation of "trying to be liked" varies significantly depending on the gender of both the speaker and the interlocutor.

Acoustic features associated with the perception of 'trying to be liked' across different gender pairings.

As shown in the data, men generally expand their pitch range when attempting to be liked. However, women exhibit more varied strategies: they raise their pitch when speaking to men but lower it when interacting with other women. This suggests that social speech is not a static set of features but a dynamic adaptation to the social environment.

The Backchannel Paradox and Turn-Taking

The research also highlights a "Backchannel Paradox." While backchannels (e.g., "uh-huh," "okay") are traditionally viewed as positive indicators of engagement, the study found that excessive backchanneling can negatively impact a speaker's likeability to outside observers.

Distribution of turn-taking behaviors based on speaker role and relative likeability.

Specifically, "Followers" who produced a high volume of backchannels were liked significantly less by raters compared to their interlocutors (110 vs 28 instances). In contrast, smooth turn-taking—characterized by non-disruptive switches and minimal silent gaps—correlated with higher perceived social competence and likeability. Followers who overlapped their speech slightly with the Describer were generally perceived more favorably than those who left silent gaps.

Perceived Planning and Likeability

Perceptions of "speech planning" were found to correlate with longer intonational phrases and slower speaking rates. Interestingly, the presence of filled pauses ("um," "uh") did not necessarily detract from the speaker's image; instead, they served as markers that the speaker was actively planning their contribution rather than simply making mistakes.

General likeability was associated with higher intensity (volume), lower pitch, and lower shimmer (voice stability). Lexically, liked speakers tended to use more "active" and "image-heavy" language, suggesting that vocal confidence and descriptive clarity are key drivers of positive social perception.

Conclusion

This study demonstrates that social behaviors are encoded in complex acoustic and prosodic patterns that are sensitive to gender and conversational role. By bridging the gap between subjective human judgment and objective speech features, this work provides a foundation for developing more socially aware speech technologies. Future research will aim to incorporate additional social variables, such as dominance and engagement, to further refine these models.

Summary updated

BibTeX
@inproceedings{gravano2011acoustic,
  author = {Agustín Gravano and Rivka Levitan and Laura Willson and Stefan Benus and Julia Hirschberg and Ani Nenkova},
  title = {{Acoustic and Prosodic Correlates of Social Behavior}},
  booktitle = {Interspeech},
  year = {2011},
  doi = {10.21437/interspeech.2011-25},
}

Loading related content...

Figure