Disambiguating the functions of conversational sounds with prosody: the case of 'yeah'

Khiet P. Truong, Dirk HeylenInterspeech 2010

TL;DR

This paper explores how the prosody (pitch, duration, intensity) of the common word "yeah" can help distinguish its different conversational functions, like acting as a simple acknowledgment versus signaling an intent to take the conversational floor. The authors demonstrate that prosodic features, when combined with conversational state information, significantly improve the accuracy of classifying these functions, offering valuable insights for human-computer interaction and social signal processing.

What Problem Does This Paper Address?

Conversational speech is rich with subtle cues that convey meaning beyond the literal words spoken. The word "yeah," despite its apparent simplicity, is a prime example of this complexity. It can function as a "backchannel" – a signal that the listener is paying attention and the speaker can continue – or as an "assessment" – an evaluation of what was just said. Crucially, "yeah" can also indicate "speaker incipiency," a speaker's readiness or intention to take over the conversation. Automatically distinguishing between these functions is challenging because the same word can carry such different meanings. This ambiguity makes it difficult for machines to accurately interpret human conversation, hindering the development of more natural and responsive human-computer interfaces, such as embodied conversational agents (ECAs). This paper aims to bridge this gap by investigating how prosodic features can help disambiguate these varied functions of "yeah."

Methods

The researchers employed a decision tree (C4.5 algorithm) for classification, a common and interpretable machine learning technique. They utilized two main sets of features:

  • Conversational State (CS) Features: These features capture the immediate context of the conversation, such as whether the "yeah" speaker is also speaking in the next conversational turn, or if other speakers are talking simultaneously. They also considered "past" CS features, which rely only on historical conversational states, as these are more readily available for real-time analysis.
  • Prosodic Features: These are acoustic characteristics of the speech signal itself. The paper extracted features related to:
  • Duration: The length of the "yeah" utterance.
  • Mean Pitch and Intensity: The average pitch and loudness over the entire "yeah" utterance.
  • Pitch Contour Shape: This was captured by fitting polynomial curves (degree 1 and 2) to pitch values, aiming to represent rising, falling, or complex pitch patterns.
  • Pitch Slope: The rate of change in pitch, particularly in the latter half of the "yeah" utterance.

The experiments involved training classifiers to distinguish between different dialogue act (DA) functions of "yeah" (backchannel vs. assessment vs. other) and to differentiate between high and low speaker incipiency.

Datasets

The study used data from the AMI Corpus, a large multimodal dataset comprising 100 hours of meeting recordings. For this specific research, they focused on turn-initial "yeah" utterances from 14 meetings, resulting in a dataset of 1034 "yeah" instances. This dataset was annotated for dialogue acts, providing the ground truth for their classification tasks.

Key Insights and Ideas

  • "Yeah" is more than just "yes": The paper strongly reinforces that "yeah" is a highly versatile word with distinct communicative functions that go beyond simple agreement.
  • Prosody matters for "yeah": The study provides empirical evidence that prosodic features are not just decorative but actively contribute to conveying the intended meaning of "yeah" in conversation.
  • Duration is a key differentiator: The length of a "yeah" utterance emerged as a particularly important prosodic feature for distinguishing its function, with assessments generally being longer than backchannels.
  • Speaker incipiency is subtle but detectable: While the prosodic contribution to predicting speaker incipiency was less pronounced than for DA function, it still offered additive value, suggesting that subtle acoustic cues can signal a speaker's intent to take the floor.
  • "Yeah" signals greater intent than "mm/mm-hmm": The research confirms previous findings that "yeah" is more likely to be followed by the same speaker initiating a new turn compared to other continuers like "mm" or "uh-huh."

Key Results

  • Disambiguating DA functions: When using conversational state features alone, the AUC for classifying "yeah" into backchannel, assessment, or other categories was 0.69. Adding prosodic features, particularly duration (dur_z), significantly improved this to 0.67 (for bck vs ass vs oth) and 0.70 (for bck vs ass), demonstrating the added value of prosody.
  • Speaker Incipiency: For distinguishing high vs. low speaker incipiency, the baseline AUC with CS features was 0.55. Adding prosodic features improved this to 0.58, though this increase was not statistically significant. However, dur_z was identified as a useful feature, with highly incipient "yeah"s generally being longer.
  • Quantitative differences: Backchannels were found to be shorter and have lower intensity than assessments and other DAs.

Limitations and Future Work

  • Automatic Prosody Analysis Limitations: The authors acknowledge that their prosodic analyses were performed automatically, and slight inaccuracies in word boundaries from the AMI corpus transcriptions might have affected the results.
  • Definition of Speaker Incipiency: The definition of speaker incipiency used in the study was a "mechanical" one based on conversational state counts, which the authors suggest is open to discussion.
  • Future Directions:
    • Investigate other multimodal markers (beyond audio) for speaker incipiency.
  • Explore more sophisticated prosodic feature extraction, especially for pitch contours, as previous research suggested their importance.
  • Apply these findings to the design of Embodied Conversational Agents (ECAs) to make them more responsive and natural in their turn-taking and acknowledgment behaviors.

Summary updated

BibTeX
@inproceedings{truong2010disambiguating,
  author = {Khiet P. Truong and Dirk Heylen},
  title = {{Disambiguating the functions of conversational sounds with prosody: the case of 'yeah'}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure