Using responsive prosodic variation to acknowledge the user's current state

Nigel G. Ward, Rafael Escalante-RuizInterspeech 2009

TL;DR

Spoken dialog systems often sound robotic because they don't vary their prosody. This paper introduces a system that dynamically adjusts the pitch, tone, and rhythm of its acknowledgments based on the user's inferred emotional state and performance, making interactions feel more natural and preferred by users.

What Problem Does This Paper Address?

Current spoken dialog systems, particularly those used in educational or tutorial settings, tend to deliver their responses with a monotonous, unchanging prosody. This lack of vocal variation makes them feel inflexible and robotic, hindering the natural flow of human-like conversation. While prosody is known to convey a wealth of information in human-human interaction – signaling emotions, managing turn-taking, and indicating understanding – its potential in dialog systems has been largely untapped. The authors aim to bridge this gap by exploring how prosodic variation in system acknowledgments can be used to acknowledge and respond to a user's current state, thereby improving the user experience.

Methods

The researchers adopted a bottom-up approach, analyzing an existing corpus of memory quiz dialogs to identify how prosody was used expressively. They identified eleven dimensions of prosodic variation and their associated functions, such as signaling encouragement, indicating a need to slow down, or conveying confidence.

They then developed rules to detect situations where specific prosodic variations would be appropriate, based on factors like the delay between the user's guess and the system's acknowledgment, the user's recent performance (number of hints, incorrect guesses), and the pitch contour of the user's last utterance.

These rules were implemented in a "Wizard of Oz" testbed. Two versions of the system were compared: a baseline system with randomly varied prosody and a rule-based system that dynamically adjusted prosody according to the identified rules. The prosodic variations were realized using various audio manipulation techniques on a neutral "Good job" utterance.

Datasets

The study utilized a corpus of 48 memory quiz dialogs between a master tutor and 16 students, previously collected and described in [2]. This corpus was used for the initial analysis of prosodic variation and the development of the rules. No new datasets were introduced.

Key Insights and Ideas

  • Prosody is a rich communication channel: Even subtle variations in pitch, rhythm, and tone carry significant pragmatic meaning, influencing user perception of the system's responsiveness and empathy.
  • Context is king for prosody: The same prosodic feature can convey different meanings depending on the dialog context. For example, a condescending tone in one situation might signal empathy in another where the user is struggling.
  • User state matters: A system's ability to infer and respond to a user's current state (e.g., confidence, difficulty, pace) through prosody can significantly enhance the interaction.
  • Bottom-up analysis is effective: Analyzing real-world dialogs to discover prosodic functions and then building rules based on these observations can lead to practical and effective system designs.
  • Prosody can be more impactful than lexical choice: The authors suggest that varying prosody might be a more straightforward and effective way to improve dialog system responsiveness compared to varying the words used.

Key Results

The rule-based system, which dynamically adjusted acknowledgment prosody based on user state, was preferred by users over a baseline system with random prosodic variation. While not all differences were statistically significant, users rated the rule-based system as more natural and preferred it overall. Qualitative feedback indicated that users appreciated the subtle appropriateness of the acknowledgments, even without consciously identifying the specific prosodic variations.

Limitations and Future Work

The authors acknowledge that the implemented rules are likely not exhaustive and may have limited generality. They also note that the experimental setup involved a fixed response time, which might not fully reflect real-world scenarios where speech recognition delays can occur. Future work could involve developing more sophisticated, dimensional models of prosody and user state, and exploring the application of these principles in more complex dialog domains, such as commercial interactions. The study also suggests that further research into the dynamics of affective interaction on short time scales is warranted.

Summary updated

BibTeX
@inproceedings{ward2009using,
  author = {Nigel G. Ward and Rafael Escalante-Ruiz},
  title = {{Using responsive prosodic variation to acknowledge the user's current state}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure