Positively transitioned sentiment dialogue corpus for developing emotion-affective open-domain chatbots

Weixuan Wang, Wei Peng, Chong Hsuan Huang, Haoran WangarXiv 2022 · OpenReview

Abstract: In this paper, we describe a data enhancement method for developing Emily, an emotion-affective open-domain chatbot. The proposed method is based on explicitly modeling positively transitioned (PT) sentiment data from multi-turn dialogues. We construct a dialogue corpus with PT sentiment data and will release it for public use. By fine-tuning a pretrained dialogue model using the produced PT-enhanced dialogues, we are able to develop an emotion-affective open-domain chatbot exhibiting close-to-human performance in various emotion-affective metrics. We evaluate Emily against a few state-of-the-art (SOTA) open-domain chatbots and show the effectiveness of the proposed approach.

Introduction

Most open-domain chatbots are trained on massive datasets of human conversation without regard for the emotional outcome of the interaction. This often results in "negative loops" where negative user emotions elicit negative or generic responses. To address this, researchers at Huawei developed Emily, a chatbot trained specifically to facilitate positive emotional transitions in users. By focusing on dialogues where sentiment shifts from negative or neutral to positive, Emily learns to provide more empathetic and supportive responses.

Methodology

The researchers developed a multi-step data enhancement pipeline to extract "positively transitioned" (PT) dialogue turns from unlabeled social media data.

The data enhancement pipeline for extracting positively transitioned (PT) dialogue data from raw sources.

The pipeline utilizes a Sentiment Predictor (SP) based on BERT and an Emoji-Sentiment Mapping (ES-mapper) heuristic. An utterance is only labeled if both methods agree, significantly reducing noise. From the resulting data, the team filtered for turns where the user’s sentiment moved toward a more positive state (e.g., Negative to Neutral or Neutral to Positive). This process yielded a specialized PT Sentiment Corpus of 67,205 dialogues.

Key Insights: Emotional Transitions

The motivation for focusing on transitions came from analyzing existing datasets like IEMOCAP and MELD. Analysis showed that emotional states tend to persist in natural conversation; for instance, a frustrated state often leads to further frustration.

Transition probabilities between emotional states in the IEMOCAP dataset, illustrating the tendency for emotional states to persist.

To break these cycles, the model must explicitly study the "outlier" cases where sentiment actually changes for the better. By fine-tuning DialoGPT-large on this targeted PT-enhanced data, the researchers prioritized emotional quality over sheer data volume.

Results and Evaluation

Emily was evaluated against state-of-the-art models including Meta’s Blender and Baidu’s PLATO-2 using several automated metrics:

  • Context Correlation: Emily achieved a score of 0.4401, significantly higher than Blender (0.3098) and PLATO-2 (0.3918), indicating higher relevance to emotional context.
  • Fluency: Emily (0.3398) outperformed both SOTA models, approaching the human reference score of 0.3595.
  • Qualitative Performance: In sample dialogues, Emily demonstrated a superior capacity for validating user feelings and offering supportive, constructive responses compared to baseline models.

Conclusion

This work demonstrates that explicitly modeling positive sentiment transitions is an effective strategy for building emotion-affective chatbots. The release of the PT Sentiment Corpus provides a valuable resource for the research community to further develop empathetic AI systems.

Reference: Wang, W., Peng, W., Huang, C. H., & Wang, H. (2022). Positively transitioned sentiment dialogue corpus for developing emotion-affective open-domain chatbots. arXiv:2208.04565v2 [cs.CL].

Summary updated

Abstract

In this paper, we describe a data enhancement method for developing Emily, an emotion-affective open-domain chatbot. The proposed method is based on explicitly modeling positively transitioned (PT) sentiment data from multi-turn dialogues. We construct a dialogue corpus with PT sentiment data and will release it for public use. By fine-tuning a pretrained dialogue model using the produced PT-enhanced dialogues, we are able to develop an emotion-affective open-domain chatbot exhibiting close-to-human performance in various emotion-affective metrics. We evaluate Emily against a few state-of-the-art (SOTA) open-domain chatbots and show the effectiveness of the proposed approach. The corpus is made publicly available.
BibTeX
@misc{wang2022positively,
  author = {Weixuan Wang and Wei Peng and Chong Hsuan Huang and Haoran Wang},
  title = {{Positively transitioned sentiment dialogue corpus for developing emotion-affective open-domain chatbots}},
  year = {2022},
  eprint = {2208.04565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.04565},
}
Figure