Abstract: In this paper, we describe a data enhancement method for developing Emily, an emotion-affective open-domain chatbot. The proposed method is based on explicitly modeling positively transitioned (PT) sentiment data from multi-turn dialogues. We construct a dialogue corpus with PT sentiment data and will release it for public use. By fine-tuning a pretrained dialogue model using the produced PT-enhanced dialogues, we are able to develop an emotion-affective open-domain chatbot exhibiting close-to-human performance in various emotion-affective metrics. We evaluate Emily against a few state-of-the-art (SOTA) open-domain chatbots and show the effectiveness of the proposed approach.
Introduction
Most open-domain chatbots are trained on massive datasets of human conversation without regard for the emotional outcome of the interaction. This often results in "negative loops" where negative user emotions elicit negative or generic responses. To address this, researchers at Huawei developed Emily, a chatbot trained specifically to facilitate positive emotional transitions in users. By focusing on dialogues where sentiment shifts from negative or neutral to positive, Emily learns to provide more empathetic and supportive responses.
Methodology
The researchers developed a multi-step data enhancement pipeline to extract "positively transitioned" (PT) dialogue turns from unlabeled social media data.
The pipeline utilizes a Sentiment Predictor (SP) based on BERT and an Emoji-Sentiment Mapping (ES-mapper) heuristic. An utterance is only labeled if both methods agree, significantly reducing noise. From the resulting data, the team filtered for turns where the user’s sentiment moved toward a more positive state (e.g., Negative to Neutral or Neutral to Positive). This process yielded a specialized PT Sentiment Corpus of 67,205 dialogues.
Key Insights: Emotional Transitions
The motivation for focusing on transitions came from analyzing existing datasets like IEMOCAP and MELD. Analysis showed that emotional states tend to persist in natural conversation; for instance, a frustrated state often leads to further frustration.
To break these cycles, the model must explicitly study the "outlier" cases where sentiment actually changes for the better. By fine-tuning DialoGPT-large on this targeted PT-enhanced data, the researchers prioritized emotional quality over sheer data volume.
Results and Evaluation
Emily was evaluated against state-of-the-art models including Meta’s Blender and Baidu’s PLATO-2 using several automated metrics:
- Context Correlation: Emily achieved a score of 0.4401, significantly higher than Blender (0.3098) and PLATO-2 (0.3918), indicating higher relevance to emotional context.
- Fluency: Emily (0.3398) outperformed both SOTA models, approaching the human reference score of 0.3595.
- Qualitative Performance: In sample dialogues, Emily demonstrated a superior capacity for validating user feelings and offering supportive, constructive responses compared to baseline models.
Conclusion
This work demonstrates that explicitly modeling positive sentiment transitions is an effective strategy for building emotion-affective chatbots. The release of the PT Sentiment Corpus provides a valuable resource for the research community to further develop empathetic AI systems.
Reference: Wang, W., Peng, W., Huang, C. H., & Wang, H. (2022). Positively transitioned sentiment dialogue corpus for developing emotion-affective open-domain chatbots. arXiv:2208.04565v2 [cs.CL].
Summary updated