Building a Dialogue Corpus Annotated with Expressed and Experienced Emotions

Tatsuya Ide, Daisuke KawaharaACL 2022

TL;DR

This paper introduces a novel Japanese dialogue corpus annotated with both "expressed" (speaker's intended emotion) and "experienced" (listener's felt emotion) emotions, addressing the limitations of existing corpora. They demonstrate that distinguishing these two emotion types is crucial for building more human-like dialogue systems, as recognizing experienced emotions is significantly harder than expressed ones.

What Problem Does This Paper Address?

Developing truly human-like dialogue systems requires them to not only understand and respond to user input but also to grasp the emotional nuances of the conversation. Humans naturally recognize an interlocutor's emotion and respond with empathy or comfort. While current research has focused on emotion recognition in text, existing dialogue corpora often annotate only the speaker's expressed emotion or the dialogue as a whole, failing to capture the dynamic emotional exchange between participants. This gap hinders the development of systems that can truly understand and generate emotionally appropriate responses, especially in multi-turn conversations where emotions evolve.

Methods

The authors propose a method to build a dialogue corpus annotated with two distinct emotion categories:

  • Expressed Emotion: The emotion a speaker intends to convey through their utterance.
  • Experienced Emotion: The emotion a listener feels upon hearing the utterance.

They collected dialogues from Twitter, filtering for text-only interactions and limiting dialogues to a maximum of nine turns to maintain manageable complexity. The crucial step was the crowdsourcing annotation process. They used Plutchik's wheel of emotions as the label set and asked crowdworkers to annotate each utterance for both expressed and experienced emotions. Importantly, they allowed for multiple emotion labels per utterance and considered their intensity (strong/weak) based on worker consensus. This approach allows for a more granular understanding of emotional states and their transitions.

Datasets

The paper introduces a new Japanese emotion-tagged dialogue corpus.

  • Source: Collected from Twitter.
  • Size: 3,828 dialogues and 13,806 utterances.
  • Annotation: Each utterance is annotated with expressed and experienced emotions, using Plutchik's eight emotion labels (anger, anticipation, joy, trust, fear, surprise, sadness, disgust) with intensity levels (strong/weak).

This corpus differs from existing ones by explicitly annotating both expressed and experienced emotions at the utterance level, providing a richer dataset for studying emotional dynamics in dialogue.

Key Insights and Ideas

  • Expressed vs. Experienced Emotions are Distinct: Statistical analysis and experimental results clearly show that recognizing experienced emotions is more challenging than expressed emotions, highlighting the importance of annotating both.
  • Multi-task Learning is Beneficial: Training a single model to recognize both expressed and experienced emotions simultaneously improves performance on both tasks, suggesting a shared underlying signal or complementary information.
  • Emotional Trajectories in Dialogue: Analysis of emotion changes from the beginning to the end of dialogues reveals that speakers often tend towards positive emotions (anticipation, joy) regardless of their initial state, suggesting a potential "comforting" or "positive-seeking" nature of extended conversations.
  • Granularity Matters: Annotating emotion intensity (strong/weak) provides a more nuanced view and is crucial for understanding the subtle shifts in emotional states.

Key Results

  • Emotion Recognition Difficulty: Models achieved lower correlation coefficients for experienced emotions compared to expressed emotions across all tested models (e.g., NICT model: 56.23/43.88 for expressed/experienced).
  • Multi-task Learning Improvement: Multi-task learning models consistently outperformed single-task models, showing an improvement of about one point in Pearson's and Spearman's correlation coefficients for both expressed and experienced emotion recognition tasks.
  • Emotion Label Performance: Anticipation and Joy were the easiest emotions to recognize for both expressed and experienced categories, while Trust and Fear (expressed) and Anger and Disgust (experienced) were the most difficult.
  • Dialogue Dynamics: Confusion matrices indicated that speakers often express anticipation and joy by the end of a dialogue, even if they started with other emotions.

Limitations and Future Work

  • Annotation Subjectivity: While they used multiple annotators to ensure reliability, the "expressed emotion" annotation is still what workers think a speaker would want to express, not necessarily their true internal state. True subjective annotation for expressed emotion is a future goal.
  • Dialogue Context: The current annotation focuses on utterances and their immediate context. Future work should consider broader dialogue situations, topics, and speaker intentions for a more comprehensive understanding of emotion.
  • Response Generation: The authors aim to use their corpus to build dialogue systems that can predict experienced emotions and generate appropriate expressed emotions in response, moving beyond recognition to generation.
  • Cross-lingual Applicability: While the method is proposed for Japanese, it can be applied to any language, suggesting potential for broader corpus development.

Summary updated

BibTeX
@inproceedings{ide2022building,
  author = {Tatsuya Ide and Daisuke Kawahara},
  title = {{Building a Dialogue Corpus Annotated with Expressed and Experienced Emotions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure