Emotion Inference in Multi-Turn Conversations with Addressee-Aware Module and Ensemble Strategy

Dayu Li, Xiaodan Zhu, Yang Li, Suge Wang, Deyu Li, Jian Liao, Jianxing ZhengEMNLP 2021 · OpenReview

Introduction

Conversational AI has made significant strides in understanding human language, but emotional intelligence remains a frontier. While most research focuses on Emotion Recognition in Conversation (ERC)—identifying how someone feels about what they just said—this EMNLP 2021 paper tackles a more proactive challenge: Emotion Inference. The goal is to predict how a participant will feel in their next turn before they even speak.

The Challenge of Proactive Emotion Prediction

Traditional ERC models are retrospective; they label emotions after the fact. For a chatbot to be truly effective in dialogue planning or emotional support, it must anticipate the emotional impact of its own words or the words of others. This task is difficult for two main reasons:

  1. Missing Information: The model lacks the text of the future turn to analyze.
  2. Complex Emotional Dynamics: Human emotions are influenced by persistence (the tendency to stay in the same mood) and contagiousness (the tendency to mirror the emotions of others).

In multi-party conversations, such as those in the Friends TV show (used in the MELD and EmoryNLP datasets), tracking these influences becomes complex as multiple speakers interact with the "addressee."

Emotional states in datasets like MELD can be intense and varied, requiring models to anticipate sudden shifts.

Methodology: The DialogInfer Framework

The researchers proposed two architectures, DialogInfer-S (sequence-based) and DialogInfer-G (graph-based), both utilizing a novel Addressee-Aware Module.

The Addressee-Aware Module

This module acts as a traffic controller for emotional information, distinguishing between internal and external influences:

  • Persistence: When the addressee was the last speaker, the model uses specialized units ($LSTM_{store}$ or $ATT_{store}$) to capture their internal emotional state.
  • Contagiousness: When someone else speaks, different units ($LSTM_{affect}$ or $ATT_{affect}$) model how this external input might alter the addressee's mood.

Ensemble Strategy and Fusion

Because human emotional reactions are not deterministic, the authors employed an ensemble strategy. They generated 15 candidate results using different random seeds and then used a fusion classifier to select the most likely outcome based on the conversation context and the pool of candidates.

Experimental Results

The models were evaluated on three major benchmarks: IEMOCAP (dyadic), MELD (multi-party), and EmoryNLP (multi-party).

Predicting a return to a neutral or positive state is equally important for maintaining conversational flow.

Key findings include:

  • State-of-the-Art Performance: The RoBERTa-based ensemble reached a weighted macro-F1 of 66.39 on IEMOCAP, significantly outperforming previous methods like DialogueGCN and COSMIC.
  • Importance of Speaker Identity: Ablation studies showed that removing the addressee-aware module (treating all speakers identically) caused performance to drop sharply (e.g., from 66.39 to 61.16 on IEMOCAP). This proves that distinguishing between "self-influence" and "other-influence" is critical.
  • Contextual Depth: Simple models like RoBERTa Large that only consider the current turn performed poorly (~43 F1 on IEMOCAP), highlighting that emotion inference is a long-term temporal task.

Conclusion and Future Directions

The "DialogInfer" framework demonstrates that explicitly modeling emotional persistence and contagiousness is key to predicting future states. While predicting the future remains harder than labeling the past—especially in complex multi-party settings—this work lays the groundwork for more empathetic and proactive AI. Future work may integrate multimodal cues (audio/video) and commonsense knowledge to bridge the "future gap" further.

Summary updated

Abstract

Emotion inference in multi-turn conversations aims to predict the participant’s emotion in the next upcoming turn without knowing the participant’s response yet, and is a necessary step for applications such as dialogue planning. However, it is a severe challenge to perceive and reason about the future feelings of participants, due to the lack of utterance information from the future. Moreover, it is crucial for emotion inference to capture the characteristics of emotional propagation in conversations, such as persistence and contagiousness. In this study, we focus on investigating the task of emotion inference in multi-turn conversations by modeling the propagation of emotional states among participants in the conversation history, and propose an addressee-aware module to automatically learn whether the participant keeps the historical emotional state or is affected by others in the next upcoming turn. In addition, we propose an ensemble strategy to further enhance the model performance. Empirical studies on three different benchmark conversation datasets demonstrate the effectiveness of the proposed model over several strong baselines.
BibTeX
@inproceedings{li2021emotion,
  author = {Dayu Li and Xiaodan Zhu and Yang Li and Suge Wang and Deyu Li and Jian Liao and Jianxing Zheng},
  title = {{Emotion Inference in Multi-Turn Conversations with Addressee-Aware Module and Ensemble Strategy}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=SZ1TAW9gbS},
}
Figure