Responding to user emotional state by adding emotional coloring to utterances

Jaime C. Acosta, Nigel G. WardInterspeech 2009

TL;DR

This paper explores how to make spoken dialog systems more engaging and persuasive by enabling them to respond to a user's emotional state. The authors collected and analyzed persuasive dialogs, finding that human speakers adapt their emotional tone in response to their interlocutor, and developed a predictive model to mimic this behavior in AI.

What Problem Does This Paper Address?

Current spoken dialog systems are largely limited to simple information exchange and lack the ability to engage in more complex interactions like persuasion or guidance. A key missing piece is the ability to understand and respond to the user's emotional state, which is crucial for building rapport and making interactions more effective. Human conversations are rich with non-verbal cues, including emotional coloring in speech (prosody, pitch, timing, volume), which humans instinctively pick up on and respond to. This paper aims to bridge this gap by enabling AI to detect and react to user emotions, specifically in the context of persuasive dialogs.

Methods

The researchers collected a corpus of persuasive dialogs between a graduate school coordinator and undergraduate students. They then employed human judges to label each utterance along three emotional dimensions: activation (active/passive), valence (positive/negative), and power (dominant/submissive). These labels were on a continuous scale from -100 to +100.

To understand how the coordinator responded to student emotions, they analyzed "adjacency pairs" (a student utterance followed by a coordinator response). They computed correlation coefficients to see if the coordinator's emotional dimensions mirrored or inversely related to the student's.

Finally, they built a predictive model using machine learning (specifically REPTrees and Bagging with REPTrees) to predict the coordinator's emotional response based on the student's emotional state. The student's emotional dimensions (activation, valence, power) served as attributes for the model.

Datasets

The paper introduces a newly collected corpus of persuasive dialogs. This corpus consists of six dialogs, each lasting 9-20 minutes, between a graduate school coordinator and 10 undergraduate students. The dialogs were focused on persuading students to consider graduate school.

Key Insights and Ideas

  • Emotional Mirroring is Key: The study found that human speakers, particularly the coordinator in this corpus, often mirror the emotional valence of their interlocutor. This suggests that acknowledging and reflecting a user's feelings is a fundamental aspect of persuasive and rapport-building dialog.
  • Power Dynamics Matter: There's an inverse relationship in the "power" dimension: when a student sounds more dominant, the coordinator tends to become more submissive, and vice versa. This reflects the natural ebb and flow of conversation where one person might lead while the other yields.
  • Prosody as a Rich Signal: The emotional coloring of speech (pitch, timing, volume) is not random but serves as a primary way for speakers to convey attention, involvement, and empathy. Systems lacking this emotional dimension feel less like real interactions and more like browsing audio clips.
  • Predicting Emotional Responses is Possible: The research demonstrates that it's feasible to build predictive models that can anticipate a system's appropriate emotional response based on the user's expressed emotions in the previous utterance.

Key Results

  • Valence Mirroring: A significant positive correlation (0.34) was found between the student's and coordinator's valence dimensions, indicating clear emotional mirroring in positive and negative feelings.
  • Power Inverse Relationship: A significant negative correlation (-0.31) was observed between the student's and coordinator's power dimensions, suggesting a dynamic give-and-take in conversational control.
  • Predictive Model Performance: The machine learning models achieved promising correlation coefficients (up to 0.34 for power, 0.28 for activation, and 0.35 for valence) in predicting the coordinator's emotional dimensions based on the student's. This suggests that even with limited features, emotional adaptation can be partially modeled.

Limitations and Future Work

  • Data Quality and Variability: Poor recording quality, overlapping speech, and very short utterances were identified as factors that negatively impacted the predictive model's accuracy. The first dialog collected also showed higher errors, possibly due to the coordinator still developing her strategies and the student having a distinct speaking style.
  • Beyond Immediate Context: The authors acknowledge that the coordinator's responses were likely influenced by more than just the immediate previous utterance. Factors like the student's personality, gender, age, social status, and the cumulative interpretation of the user's state probably played a role.
  • Lexical Influence: The influence of the student's lexical content on the coordinator's emotional coloring was not explicitly modeled but is considered a potential factor.
  • Future Directions:
    • Finer-grained Modeling: Exploring models that go beyond utterance-by-utterance analysis to capture more immediate dependencies within shorter time periods.
  • Cross-Domain Validation: Testing whether the identified emotional strategies are applicable to other dialog domains and types.
  • Adequacy of Dimensions: Investigating if the three chosen emotional dimensions (activation, valence, power) are sufficient to capture all significant prosodic variations.
  • Rapport Building: The ultimate goal is to build a system that can detect emotions automatically and use this to generate emotionally colored responses, aiming to build rapport with users. This involves integrating emotion detection, predictive modeling, and emotional speech synthesis.

Summary updated

BibTeX
@inproceedings{acosta2009responding,
  author = {Jaime C. Acosta and Nigel G. Ward},
  title = {{Responding to user emotional state by adding emotional coloring to utterances}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure