Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum

Wei Ai, Fuchen Zhang, Yuntao Shou, Tao Meng, Haowen Chen, Keqin LiAAAI 2025 · DOI

TL;DR

This paper introduces GS-MCC, a novel framework for multimodal emotion recognition in conversations that leverages graph spectrum analysis to better capture long-distance dependencies and complementary information across text, audio, and visual modalities. By employing Fourier graph operators and contrastive learning, GS-MCC effectively addresses the over-smoothing issue in GNNs and improves emotion recognition accuracy.

What Problem Does This Paper Address?

Multimodal Emotion Recognition in Conversation (MERC) aims to understand the emotional state of each utterance in a dialogue by integrating information from text, audio, and visual cues. While existing methods often rely on Transformers or Graph Neural Networks (GNNs), they struggle with two key challenges:

  1. Insufficient Long-Distance Dependence Perception: Traditional GNNs, due to their inherent over-smoothing or low-pass filtering characteristics, have difficulty capturing emotional dependencies that span across many utterances in a conversation. This is often exacerbated by the use of sliding windows, which limit the context.
  2. Underutilization of High-Frequency Features: GNNs tend to aggregate neighborhood information, focusing on consistent (low-frequency) features while suppressing dissimilar (high-frequency) features. However, both types of features are crucial for accurate emotion recognition, especially when one modality's signal is weak and needs to be complemented by another. Existing attempts to fuse high and low-frequency features have not fully explored their collaborative potential.

Methods

The proposed Graph-Spectrum-based Multimodal Consistency and Complementary collaborative learning framework (GS-MCC) tackles these challenges through several key modules:

  • Multimodal Interaction Graph Construction: A sliding window approach is used to build a graph where each utterance's modalities (text, audio, vision) are nodes. Edges represent both intra-modal relationships within the window and inter-modal interactions within an utterance.
  • Fourier Graph Operators (FGO): To overcome the limitations of traditional GNNs, GS-MCC employs FGOs. These operators work in the frequency domain, allowing for efficient extraction of both low-frequency (consistent) and high-frequency (complementary) semantic information. By stacking multiple layers of FGOs, the framework can effectively capture long-distance dependencies and mitigate the over-smoothing problem.
  • Contrastive Learning for Feature Collaboration: GS-MCC utilizes a contrastive learning strategy to promote synergy between the extracted high- and low-frequency features. It constructs self-supervised signals that encourage consistent and complementary semantic learning. Specifically, it employs two contrastive losses:
  • Low-Frequency Contrastive Learning (LFCL): Uses low-frequency samples as anchors and high-frequency samples as negatives.
  • High-Frequency Contrastive Learning (HFCL): Uses high-frequency samples as anchors and low-frequency samples as negatives. This approach aims to increase the frequency domain difference between contrasting views, leading to better contrast learning effects.

  • Emotion Classifier: The final embeddings, obtained by combining the time-domain representations of high and low-frequency features, are fed into an MLP network and a softmax function for emotion prediction.

Datasets

The paper evaluates GS-MCC on two widely-used benchmark multimodal datasets for emotion recognition:

  • IEMOCAP (Interactive emotional dyadic motion capture database): A well-known dataset for multimodal emotion recognition.
  • MELD (Multimodal Multy-Party Dataset for Emotion Recognition in Conversations): Another standard benchmark dataset.

Key Insights and Ideas

  • Graph Spectrum as a Solution: Shifting the perspective to the graph spectrum provides a powerful lens for understanding and extracting different types of information (high vs. low frequency) from conversational graphs.
  • Fourier Operators for Efficiency and Depth: Fourier graph operators offer a computationally efficient way to perform graph convolutions in the frequency domain, enabling deeper GNN architectures without succumbing to over-smoothing.
  • Synergistic Contrastive Learning: The proposed LFCL and HFCL are not just about extracting features separately but about forcing them to collaborate. By contrasting across frequency domains, the model learns to better integrate complementary and consistent semantic signals.
  • Speaker Information is Crucial: Ablation studies highlight the significant impact of speaker embeddings, underscoring their importance in understanding individual emotional nuances within a conversation.

Key Results

  • GS-MCC achieves state-of-the-art performance on both IEMOCAP and MELD datasets, outperforming several strong baselines. For instance, it achieves a W-F1 score of 69.0% on MELD, a 2.2% improvement over the next best baseline (AdaIGN).
  • The model demonstrates significant improvements in individual emotion categories, such as "happy," "neutral," "excited," and "frustrated" on IEMOCAP, and "fear," "joy," and "disgust" on MELD.
  • GS-MCC exhibits a much lower parameter count (2.10M) compared to other GCN-based methods like DER-GCN (78.59M), indicating greater computational efficiency.
  • The proposed method effectively mitigates the over-smoothing phenomenon in GNNs, as shown in the analysis comparing GS-MCC with MMGCN at different layer depths.
  • Inference time is significantly reduced compared to other GCN-based methods, with GS-MCC achieving 56.8s on IEMOCAP and 71.25s on MELD.

Limitations and Future Work

  • Over-smoothing Mitigation: While GS-MCC alleviates over-smoothing, it's noted that the problem can still be present to some extent, especially with very deep networks.
  • Multimodal Feature Importance: Ablation studies show that text features are most impactful, while video features are the least effective, suggesting potential areas for improvement in video feature extraction or fusion.
  • Future Work: The paper implicitly suggests future work could explore more advanced graph spectrum techniques, novel contrastive learning strategies, and potentially investigate the role of other modalities or more complex conversational structures. The efficiency and effectiveness of GS-MCC also open doors for its application in real-time or resource-constrained scenarios.

Summary updated

BibTeX
@inproceedings{ai2025revisiting,
  author = {Wei Ai and Fuchen Zhang and Yuntao Shou and Tao Meng and Haowen Chen and Keqin Li},
  title = {{Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33242},
}
Figure