UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause

Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, Jiayuan Xie

TL;DR

UniMEEC is a novel unified framework that bridges two critical tasks in conversational AI: identifying a speaker's emotion and pinpointing the specific event that caused it. By reformulating these tasks as a single "mask prediction" problem using causal prompts, the model leverages the inherent logical link between an emotion and its trigger to achieve new state-of-the-art performance across multiple multimodal benchmarks.

What Problem Does This Paper Address?

In the field of affective computing, researchers typically tackle two distinct problems: Multimodal Emotion Recognition in Conversation (MERC)—identifying what someone is feeling (e.g., joy, anger)—and Multimodal Emotion-Cause Pair Extraction (MECPE)—identifying why they feel that way.

Historically, these have been treated as separate silos. However, human emotions don't exist in a vacuum; they are reactions to specific events. For example, if a speaker is "happy" because they were "praised," there is a clear causal relationship. The authors argue that by ignoring this causality, current models miss out on vital context that could improve accuracy for both tasks.

Figure 1: Illustration of the causal inference between emotion and emotion cause, which unifies MECPE and MERC tasks. “response” denotes the speaker’s reaction to the event and “event” denotes the event that triggers emotion.

Methods

The authors introduce UniMEEC, which stands for Unified Multimodal Emotion recognition and Emotion-Cause analysis. The architecture consists of two primary innovations:

  1. Multimodal Causal Prompt (MCP): Instead of standard classification, the authors reformulate the problem into a "fill-in-the-blanks" task for a Pre-trained Language Model (PLM) like BERT. They use a template: "The emotion of [X] is [M1], and its emotion cause is [M2]". Here, [X] is the multimodal input, [M1] is the emotion label, and [M2] is the cause utterance ID. This forces the model to learn the semantic and causal coherence between the two slots.
  2. Task-specific Hierarchical Context (THC): To handle the flow of a conversation, the model uses a 3-level Graph Attention Network (GAT). This structure aggregates information hierarchically: from the raw utterance level (bottom) to the cause-specific level (middle), and finally to the emotion-specific level (top). This directed flow helps the model filter out conversational noise and focus on relevant triggers.
Figure 2: The overview of UniMEEC. The outputs “disgust” and “u3” denote the emotion category and the emotion cause utterance ID of target utterance u6, respectively.

Datasets

The model was evaluated on four widely-used public benchmarks that include text, audio, and video modalities:

  • MERC Tasks: MELD (multi-party conversations from Friends) and IEMOCAP (dyadic emotional acting).
  • MECPE Tasks: ECF (an annotated version of MELD for causes) and ConvECPE (an annotated version of IEMOCAP).

Key Insights and Ideas

  • Causality as a Regularizer: One of the most striking findings is that adding the cause-extraction task actually improves emotion recognition. The "why" provides a logical anchor for the "what," which is particularly helpful for rare emotion categories (like "Fear" or "Disgust") where data is scarce.
  • Prompting Beyond Text: The paper demonstrates that prompt-tuning—a technique usually reserved for pure NLP—can be effectively adapted for multimodal data by injecting acoustic and visual features directly into the prompt's embedding space.
  • Hierarchical Filtering: The THC module proves that not all context is equal. By restricting the "attention" of the model to specific windows and levels, UniMEEC avoids the "distraction" of irrelevant utterances in long conversations.

Key Results

UniMEEC consistently outperformed previous state-of-the-art (SOTA) models across all datasets:

  • Emotion Recognition: On the IEMOCAP and MELD datasets, UniMEEC improved the Weighted F1 (WF1) scores by 1.99% and 2.06% respectively compared to previous leaders like MultiEMO and MALN.
  • Cause Extraction: On the ECF dataset, the model saw significant jumps in "Pair Extraction" F1 scores, outperforming the previous best by over 3%.
Table 2: Results on IEMOCAP and MELD datasets. The best results are highlighted in bold. The results with underline denote the previous SOTA performance.

Ablation studies confirmed that the "unified" nature of the task is the secret sauce: removing the MECPE (cause) component from the training led to a performance drop of up to 3.57% in emotion recognition.

Limitations and Future Work

Despite the gains, the authors acknowledge several hurdles:

  • Modality Alignment: Because audio and video features are often shorter than text sequences, the model uses zero-padding to align them. This can introduce "dead" information that might slightly dilute the fusion representations.
  • Text-Only Performance: The current framework is heavily optimized for multimodal scenarios; its effectiveness on purely textual emotion-cause tasks remains a subject for future research.
  • Real-World Readiness: While SOTA, the absolute accuracy on complex datasets like MELD remains around 68-74%, suggesting that fully understanding human conversational intent is still a work in progress for AI.

Summary updated

Abstract

Multimodal emotion recognition in conversation (MERC) and multimodal emotion-cause pair extraction (MECPE) have recently garnered significant attention. Emotions are the expression of affect or feelings; responses to specific events, or situations -- known as emotion causes. Both collectively explain the causality between human emotion and intents. However, existing works treat emotion recognition and emotion cause extraction as two individual problems, ignoring their natural causality. In this paper, we propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework (UniMEEC) to explore the causality between emotion and emotion cause. Concretely, UniMEEC reformulates the MERC and MECPE tasks as mask prediction problems and unifies them with a causal prompt template. To differentiate the modal effects, UniMEEC proposes a multimodal causal prompt to probe the pre-trained knowledge specified to modality and implements cross-task and cross-modality interactions under task-oriented settings. Experiment results on four public benchmark datasets verify the model performance on MERC and MECPE tasks and achieve consistent improvements compared with the previous state-of-the-art methods.
BibTeX
@inproceedings{hu2024unimeec,
  author = {Guimin Hu and Zhihong Zhu and Daniel Hershcovich and Lijie Hu and Hasti Seifi and Jiayuan Xie},
  title = {{UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.302},
}
Figure