Adaptive Graph Learning for Multimodal Conversational Emotion Detection

Geng Tu, Tian Xie, Bin Liang, Hongpeng Wang, Ruifeng Xu

Abstract

Emotion Recognition in Conversations (ERC) is a critical task for developing emotionally intelligent AI, yet it remains challenging due to the complex interplay of multimodal signals and conversational dynamics. Traditional models often rely on future information (future leakage) or fail to dynamically balance a speaker's internal emotional state with external influences. We present AdaIGN (Adaptive Interactive Graph Network), a framework that treats conversations as directed graphs and employs learnable selection policies to identify the most relevant modalities and contexts. AdaIGN effectively navigates the "inertia-empathy" tug-of-war, achieving state-of-the-art results on the IEMOCAP and MELD datasets.

The Challenge of Conversational Emotion Recognition

Human emotions in dialogue are rarely static; they are influenced by a combination of internal history and external social cues. The authors identify three primary hurdles in current ERC research:

  1. Future Leakage: Many models utilize "future" utterances to predict current emotions. While effective for offline analysis, this prevents real-time deployment where future data is unavailable.
  2. The Inertia-Empathy Tug-of-War: ERC requires a delicate balance between emotional inertia (the influence of a speaker’s own previous state) and empathy (the influence of the conversation partner). Most existing models treat these as static weights, failing to capture the fluid nature of human interaction.
  3. Multimodal Conflict: Modalities often provide conflicting signals—for instance, a speaker might use neutral words while exhibiting sad facial expressions. Standard fusion techniques often struggle to resolve these discrepancies.
Figure 1: An illustration of emotional inertia and empathy in a conversation between Joey and Rachel.

As shown in Figure 1, a speaker's emotion (e.g., Rachel's sadness) can persist across multiple turns (inertia) or be triggered by the partner's inquiry (empathy).

The AdaIGN Framework

To address these challenges, the authors propose the Adaptive Interactive Graph Network (AdaIGN). This model represents a conversation as a directed graph where nodes correspond to multimodal utterances and edges represent the flow of influence over time.

Figure 5: The complete architecture of AdaIGN, featuring the Utterance-level Encoder, Adaptive Interactive Graph Network, and Emotion Classifier.

Key Components:

  • Directed Interactive Graph: By using directed edges, AdaIGN ensures that only past and current information informs the prediction, making the model strictly causal and suitable for real-time applications.
  • Node and Edge-level Selection Policies (NESP): This serves as the core "decision-making" unit. Utilizing the Gumbel-Softmax trick, the model learns to dynamically "keep or drop" specific modalities (nodes) and historical influences (edges). This allows the model to prioritize the speaker's history or the partner's influence depending on the specific conversational context.
  • Graph-Level Selection Policy (GSP): This component integrates the original graph features with the policy-refined features, ensuring a robust representation for final classification.
  • Self-Supervised Learning: To improve efficiency, the model uses self-supervised pseudo-labels. When multiple modalities agree on an emotion, the model "masks" these easier cases, focusing its learning capacity on ambiguous or conflicting utterances.

Experimental Evaluation

The model was rigorously tested on two benchmark datasets:

  • IEMOCAP: A dataset of professional actor performances (7,433 utterances) across six emotion categories.
  • MELD: A multi-party dataset from the TV show Friends (13,708 utterances). MELD is notoriously difficult due to its noisy, real-world audio and complex multi-speaker dynamics.

Results and Insights

AdaIGN established new state-of-the-art (SOTA) benchmarks:

  • IEMOCAP: Achieved a weighted F1-score of 70.74%, surpassing previous models like CMCF-SRNet.
  • MELD: Achieved a weighted F1-score of 66.79%, representing a significant 4.49% improvement over previous SOTA. This highlights AdaIGN's robustness in noisy, multi-party environments.

Ablation studies confirmed that the selection policies (NSP/ESP) are vital; removing them led to a measurable performance decline. This proves that filtering information is as critical as processing it.

Conclusion and Future Directions

AdaIGN demonstrates that adaptive, directed graph learning is a powerful approach for real-time ERC. By dynamically balancing emotional inertia and empathy, it more closely mirrors human social cognition. However, challenges remain, particularly regarding class imbalance (e.g., low performance on rare emotions like "fear") and sudden emotional shifts. Future work will likely focus on improving the model's sensitivity to these abrupt transitions and leveraging more advanced self-supervised pre-training techniques.

Summary updated

Abstract

Multimodal Emotion Recognition in Conversations (ERC) aims to identify the emotions conveyed by each utterance in a conversational video. Current efforts encounter challenges in balancing intra- and inter-speaker context dependencies when tackling intra-modal interactions. This balance is vital as it encompasses modeling self-dependency (emotional inertia) where speakers' own emotions affect them and modeling interpersonal dependencies (empathy) where counterparts' emotions influence a speaker. Furthermore, challenges arise in addressing cross-modal interactions that involve content with conflicting emotions across different modalities. To address this issue, we introduce an adaptive interactive graph network (IGN) called AdaIGN that employs the Gumbel Softmax trick to adaptively select nodes and edges, enhancing intra- and cross-modal interactions. Unlike undirected graphs, we use a directed IGN to prevent future utterances from impacting the current one. Next, we propose Node- and Edge-level Selection Policies (NESP) to guide node and edge selection, along with a Graph-Level Selection Policy (GSP) to integrate the utterance representation from original IGN and NESP-enhanced IGN. Moreover, we design a task-specific loss function that prioritizes text modality and intra-speaker context selection. To reduce computational complexity, we use pre-defined pseudo labels through self-supervised methods to mask unnecessary utterance nodes for selection. Experimental results show that AdaIGN outperforms state-of-the-art methods on two popular datasets. Our code will be available at https://github.com/TuGengs/AdaIGN.
BibTeX
@inproceedings{tu2024adaptive,
  author = {Geng Tu and Tian Xie and Bin Liang and Hongpeng Wang and Ruifeng Xu},
  title = {{Adaptive Graph Learning for Multimodal Conversational Emotion Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29876},
}
Figure