TL;DR
This paper explores how to make spoken dialog systems more engaging by enabling them to detect and respond to user emotions through voice. The authors propose a system that analyzes vocal cues to infer emotional states and adapt its responses, aiming to build rapport and improve human-computer interaction.
What Problem Does This Paper Address?
As voice-based interfaces become more prevalent, a key challenge is their inability to establish "rapport" – that feeling of connection and understanding that characterizes effective human communication. While current systems excel at speech recognition and synthesis, they often lack the emotional intelligence to mirror human conversational nuances. This paper tackles the gap in creating spoken dialog systems that can infer a user's emotional state from their voice and use this information to build rapport, moving beyond purely functional interactions.
Methods
The core of this research lies in developing an emotionally intelligent spoken dialog system. The approach involves:
- Emotion Recognition: Extracting acoustic correlates of emotion from user speech. The authors focus on dimensional emotions (valence, activation, power) rather than discrete ones, believing this approach can better capture subtle emotional states.
- User Modeling: Maintaining a model of the user's current state, including their beliefs and feelings, updated based on both spoken words and inferred emotions. This allows the system to detect discrepancies, like a user saying they are "good" but sounding negative.
- Emotion Generation & Strategy: Selecting appropriate responses and vocalizations based on the user's inferred emotional state and the system's strategy for building rapport. This involves choosing content and synthesizing speech with specific emotional coloring.
- Machine Learning for Prediction: Using machine learning (Bagging with REPTrees) to predict the system's emotional responses based on the user's emotional dimensions.
Datasets
The research utilizes a persuasive dialog corpus consisting of 10 audio recordings of interactions between a graduate coordinator and individual students, totaling approximately 16 minutes of conversation per interaction. This corpus was transcribed and annotated with dimensional emotions (activation, valence, and power) by two judges.
Key Insights and Ideas
- Voice as a Rich Source of Emotion: The paper highlights that nonverbal vocal cues (pitch, speed, volume) are crucial for conveying emotion and building rapport, often more so than the literal words spoken.
- Dimensional Emotions for Nuance: Focusing on dimensions like valence, activation, and power allows for a more nuanced understanding of user emotions compared to discrete categories, which can be difficult to reliably detect.
- Emotional Mirroring for Rapport: The research suggests that mirroring a user's emotional state (e.g., matching valence) is a key strategy for building rapport, aligning with Communication Accommodation Theory.
- Detecting Contradictory States: The system's ability to identify when a user's spoken words contradict their vocal emotional expression is a significant step towards more sophisticated user understanding.
Key Results
While the paper is largely focused on the proposed system and its evaluation plan, preliminary analysis of the corpus revealed:
- Correlations in Emotional Mirroring: A correlation coefficient of 0.34 was found for emotional mirroring in the valence dimension, suggesting that coordinators tend to match the valence of student utterances.
- Inverse Relationship in Power: An inverse relationship was observed for power, where students showing more power led to coordinators showing less (-0.30 correlation coefficient).
- Machine Learning Prediction Accuracy: When predicting the coordinator's emotional responses using user emotion dimensions, correlation coefficients of 0.347 (valence), 0.344 (power), and 0.187 (activation) were achieved.
Limitations and Future Work
- User Study Pending: The full evaluation of the system's effectiveness in building rapport is planned through a user study, which has not yet been conducted.
- Focus on Voice-Only: The current research is limited to voice-only interaction, excluding other nonverbal cues like gaze or gestures that are important in embodied conversational agents.
- Future Work: The authors plan to implement and evaluate the system, testing configurations with and without emotional voice output to determine its impact on rapport. They also suggest that the learned rules for emotion detection and response could be used to train people in rapport-building skills.
Summary updated