Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

Soujanya Poria, Navonil Majumder, Rada Mihalcea, Eduard H. Hovy

Abstract

Emotion is intrinsic to humans and consequently emotion understanding is a key part of human-like artificial intelligence (AI). Emotion recognition in conversation (ERC) is becoming increasingly popular as a new research frontier in natural language processing (NLP) due to its ability to mine opinions from the plethora of publicly available conversational data in platforms such as Facebook, Youtube, Reddit, Twitter, and others. Moreover, it has potential applications in health-care systems (as a tool for psychological analysis), education (understanding student frustration) and more. Additionally, ERC is also extremely important for generating emotion-aware dialogues that require an understanding of the user's emotions. Catering to these needs calls for effective and scalable conversational emotion-recognition algorithms. However, it is a strenuous problem to solve because of several research challenges. In this paper, we discuss these challenges and shed light on the recent research in this field. We also describe the drawbacks of these approaches and discuss the reasons why they fail to successfully overcome the research challenges in ERC.

Introduction

Standard "vanilla" emotion recognition typically treats sentences in isolation. However, in real-world scenarios, the emotion behind a short utterance is often entirely dependent on the preceding context. Emotion Recognition in Conversation (ERC) aims to predict the emotion of each utterance in a dialogue by modeling complex conversational dynamics. This requires an AI to understand not just the text, but the topic, the interlocutors' personalities, their intents, and the conversational history.

This paper provides a comprehensive survey of ERC, outlining the unique challenges of modeling dialogue context, reviewing available datasets, and evaluating recent deep learning architectures designed to handle these complexities.

Modeling Conversational Dynamics

As a survey of the field, this work reviews several recent deep learning frameworks designed specifically for ERC. There has been a significant shift from standard Recurrent Neural Networks (RNNs) to more sophisticated memory-augmented and attention-based networks that can track individual speaker states.

  • Conversational Memory Networks (CMN) & Interactive Conversational Memory Networks (ICON): These early approaches utilize distinct memory banks for each speaker to model speaker-specific context and inter-speaker emotional influence.
  • DialogueRNN: A state-of-the-art method that uses a hierarchical multi-stage RNN with attention mechanisms. It explicitly tracks the speaker of the target utterance and models both self-influence (emotional inertia) and inter-speaker emotional influence over time.
  • HRLCE: A framework combining contextualized word embeddings (ELMo) with multi-head attention, primarily used for short-context conversations.
Graphical representation of conversational dynamics modeling, tracking speaker states and influences over time.

Datasets and Evaluation

The paper reviews six publicly available datasets that serve as benchmarks for ERC models:

  • IEMOCAP & SEMAINE: Multimodal datasets focusing on dyadic (two-person) interactions.
  • EmotionLines & MELD: Datasets derived from Friends TV show transcripts, with MELD providing multimodal extensions for multi-party conversations.
  • DailyDialog & EmoContext: Text-only datasets focusing on multi-turn and short-turn dialogues respectively.

Key Insights: Inertia and Shifts

A critical aspect of ERC is balancing two competing forces: emotional inertia (a speaker's tendency to maintain their current state) and inter-personal dependency (the influence speakers have on each other).

Current models are often biased toward emotional inertia. While they are reasonably good at predicting an emotion if it remains stable, they often fail when an "emotion shift" occurs due to a new stimulus. Furthermore, speaker profiling is crucial; models that track speaker-specific states perform significantly better as they can learn individual nuances and tendencies.

Example of a multi-party dialogue illustrating the complexity of emotion and sentiment tracking across multiple turns.

Results and Performance

Evaluation shows that DialogueRNN leads the pack in dyadic conversations, achieving a weighted average F1 score of 62.75% on the IEMOCAP dataset. However, the "emotion shift" bottleneck remains a significant hurdle. DialogueRNN achieves 69.2% accuracy when emotions are stable, but this drops to 47.5% when a shift occurs. Additionally, scaling to multi-party conversations (like those in the MELD dataset) remains a challenge, with state-of-the-art models showing only marginal improvements over basic baselines.

Limitations and Future Directions

The field faces several ongoing challenges. Current datasets lack annotations for emotion reasoning—the "why" behind a triggered emotion. Future systems need to move beyond recognition to reasoning. Additionally, models must become more robust to varying numbers of speakers and complex co-references in multi-party settings. Detecting conversational sarcasm and performing continuous emotion tracking in long monologues are critical next steps for achieving human-like AI understanding.

Summary updated

Abstract

Emotion is intrinsic to humans and consequently emotion understanding is a key part of human-like artificial intelligence (AI). Emotion recognition in conversation (ERC) is becoming increasingly popular as a new research frontier in natural language processing (NLP) due to its ability to mine opinions from the plethora of publicly available conversational data in platforms such as Facebook, Youtube, Reddit, Twitter, and others. Moreover, it has potential applications in health-care systems (as a tool for psychological analysis), education (understanding student frustration) and more. Additionally, ERC is also extremely important for generating emotion-aware dialogues that require an understanding of the user's emotions. Catering to these needs calls for effective and scalable conversational emotion-recognition algorithms. However, it is a strenuous problem to solve because of several research challenges. In this paper, we discuss these challenges and shed light on the recent research in this field. We also describe the drawbacks of these approaches and discuss the reasons why they fail to successfully overcome the research challenges in ERC.
BibTeX
@article{poria2019emotion,
  author = {Soujanya Poria and Navonil Majumder and Rada Mihalcea and Eduard H. Hovy},
  title = {{Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2929050},
}
Figure