Hybrid Relational Graphs with Sentiment-laden Semantic Alignment for Multimodal Emotion Recognition in Conversation

Hongru Ji, Xianghua Li, Mingxin Li, Meng Zhao, Chao GaoIJCAI 2025 · DOI

TL;DR

This paper introduces HRG-SSA, a novel framework for multimodal emotion recognition in conversations. It tackles the challenge of capturing nuanced emotional cues by proposing a hybrid relational graph that models both explicit and implicit relationships within and across modalities, enhanced by a sentiment-laden semantic alignment strategy. The approach achieves state-of-the-art performance on benchmark datasets, demonstrating the importance of richer relational modeling and sentiment integration.

What Problem Does This Paper Address?

Understanding human emotions in conversations is crucial for building more empathetic AI systems. Multimodal Emotion Recognition in Conversation (MERC) aims to achieve this by analyzing text, audio, and visual cues. While graph-based models have shown promise in capturing conversational dynamics, existing methods often focus on explicit relationships between utterances and neglect:

  • Implicit relationships: The subtle, unstated connections within a single modality (e.g., how a series of visual cues might imply a certain emotion) and across different modalities.
  • Modality-specific differences: The fact that implicit relationships can vary significantly between text, audio, and visual data.
  • Historical sentiment: The impact of past emotions on current emotional states, which is often overlooked when treating emotions as discrete labels.

This paper addresses these limitations by proposing a more comprehensive approach to modeling conversational context and emotional nuances.

Methods

The proposed HRG-SSA framework consists of an encoder and a decoder.

Encoder: * Feature Extraction: Standard techniques are used to extract features from text (T5 encoder), audio (OpenSmile), and visual (DenseNet/3D-CNN) modalities. * Hybrid Relational Graphs (HRG): This is the core novelty. It constructs a directed graph where nodes represent utterances. * Explicit Relations: Connections are made between adjacent utterances and utterances from the same speaker, preserving temporal and speaker-specific dynamics. * Implicit Connections (IC): A novel "implicit connections detector" infers relationships within each modality based on semantic similarity. This allows for capturing nuanced, non-obvious connections that explicit rules might miss. These implicit connections are modality-specific. * Graph Update: A graph attention network (GAT) framework is used to update node representations by aggregating information from neighbors, allowing for multi-hop information exchange. * Sentiment-laden Semantic Alignment (SSA): This strategy uses contrastive learning in two stages: 1. Primary Sentiment Alignment (PSA): Aligns utterance representations with their historical sentiment labels, treating correct sentiment-utterance pairs as positive and others as negative. This helps the model focus on relevant emotional cues. 2. Multimodality Semantic Alignment (MSA): Further refines cross-modal alignment by treating representations from the same modality as positive pairs and from different modalities as negative pairs. This ensures better fusion of information across modalities.

Decoder: * Text Generation: The framework treats emotion classification as a text generation task, outputting emotion labels in a fluent sentence format (e.g., "the speaker Ross's emotion is <bos_emotion>surprise <eos_emotion>"). * MultiModal Dynamic Fusion: A modified T5 decoder is used, enhanced with an additional cross-attention layer. This allows it to dynamically fuse information from the text history, the enriched graph representations, and the decoder's own state, leading to more robust emotion prediction.

Datasets

The model was evaluated on two widely used datasets for MERC:

  • IEMOCAP: Interactive emotional dyadic motion capture database.
  • MELD: Multimodal dataset for multi-party conversations, derived from the Friends TV series.

The paper notes that IEMOCAP lacks explicit emotional stance labels, so they were assigned based on existing emotional labels.

Key Insights and Ideas

  • Implicit relationships are key: The paper strongly suggests that capturing implicit connections within each modality is crucial for accurate emotion recognition, going beyond simple adjacency or speaker-based links.
  • Modality-specific graphs matter: Treating implicit relationships as modality-specific, rather than applying a uniform structure across all modalities, leads to better performance.
  • Historical sentiment enriches context: Framing emotion recognition as a text generation task and explicitly incorporating historical sentiment through contrastive learning significantly improves the model's ability to understand evolving emotional states.
  • Text generation for classification: Using a text generation approach for emotion classification allows for richer representation of emotional labels and better integration of contextual information.
  • Visual modality's complex role: While multimodal approaches are generally beneficial, the paper notes that visual information can sometimes decrease accuracy, possibly due to the complexity of scenes and the subtle nature of facial expressions within them.

Key Results

  • HRG-SSA outperforms several state-of-the-art baseline models on both IEMOCAP and MELD datasets in terms of accuracy (Acc) and weighted F1 score (w-F1).
  • On IEMOCAP, HRG-SSA achieves improvements of 6.29% in ACC and 6.48% in w-F1 compared to HAUCL.
  • On MELD, improvements are 0.34% in ACC and 0.86% in w-F1 compared to HAUCL.
  • Ablation studies confirm the significant contributions of the Hybrid Relational Graphs (HRG), Primary Sentiment Alignment (PSA), and Multimodality Semantic Alignment (MSA) modules. Removing any of these components leads to a performance drop.
  • The model performs well with text alone, but adding audio and visual modalities further enhances performance, highlighting the benefits of multimodal fusion.

Limitations and Future Work

  • Visual Modality Challenges: The authors acknowledge that the visual modality can sometimes hinder performance, suggesting future work could explore ways to better focus on relevant visual cues or handle complex scenes more effectively.
  • Interpretability: While the paper provides visualizations, further exploration into enhancing the interpretability of the model's decision-making process is suggested.
  • Adaptive Multimodal Heterogeneous Maps: The authors propose constructing adaptive multimodal heterogeneous maps as a future direction to further improve the model's ability to handle the variability of factors within each modality.

Summary updated

BibTeX
@inproceedings{ji2025hybrid,
  author = {Hongru Ji and Xianghua Li and Mingxin Li and Meng Zhao and Chao Gao},
  title = {{Hybrid Relational Graphs with Sentiment-laden Semantic Alignment for Multimodal Emotion Recognition in Conversation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/331},
}
Figure