Abstract
Multimodal Emotion Recognition in Conversation (MERC) is a critical task for developing emotionally intelligent AI. However, existing models often struggle with conflicting multimodal signals and the inability to track rapid sentiment shifts. We present GraphSmile, a framework that introduces Graph Structure Fusion (GSF) using an alternating propagation mechanism and a Sentiment Dynamics Perception (SDP) module. GraphSmile effectively integrates inter-modal and intra-modal cues while capturing the emotional trajectory of dialogues. Experimental results on IEMOCAP and MELD benchmarks demonstrate that GraphSmile achieves state-of-the-art performance, highlighting the efficacy of structured fusion and auxiliary sentiment tasks.
Introduction
Human dialogue is inherently complex, characterized by a rich interplay of textual, visual, and acoustic modalities. Current AI models for MERC face three primary challenges: incomplete contextual cues across utterances, fusion conflicts where intra-modal and inter-modal signals clash, and a lack of agility in detecting abrupt sentiment shifts. These hurdles often lead to misclassification during pivotal moments in a conversation, such as a sudden transition from neutral to angry.
Methodology
Graph Structure Fusion (GSF)
To address fusion conflicts, GraphSmile employs a novel Graph Structure Fusion (GSF) module. Unlike traditional models that process all signals simultaneously, GSF utilizes a layer-by-layer alternating approach. In the first layer, the graph propagates information exclusively between different modalities (e.g., text to video). In the subsequent layer, it propagates information within the same modality (e.g., text to text). This "inter-then-intra" cycle prevents the "over-smoothing" problem common in Graph Neural Networks and allows the model to integrate complex cues without signal interference.
Sentiment Dynamics Perception (SDP)
The Sentiment Dynamics Perception (SDP) module is a "plug-and-play" auxiliary task designed to track the emotional trajectory of a dialogue. The model is trained to predict sentiment shifts (e.g., Positive $\rightarrow$ Negative) between any two utterances. By learning these shift labels, the model gains a deeper understanding of the conversation's flow, effectively using sentiment as a regularizer to pull similar emotional states closer in the feature space.
Multitask Learning
The framework is trained using a multitask learning objective, simultaneously optimizing for nuanced emotion recognition (MERC) and coarse-grained sentiment analysis (MSAC). This allows the model to leverage the hierarchical relationship between general sentiments and specific emotional categories.
Experimental Setup and Results
GraphSmile was evaluated on four major benchmarks: IEMOCAP (6-way and 4-way), MELD, and CMU-MOSEI. The model achieved state-of-the-art results across the board:
- IEMOCAP-6: Reached an accuracy of 72.77%, outperforming previous graph-based methods like M3Net by 1.85%.
- MELD: Achieved a Weighted F1 score of 66.71%, demonstrating robustness in complex, multiparty environments.
Key Insights
- Alternation Benefits: Separating cross-modal and intra-modal processing into distinct layers yields superior results compared to simultaneous fusion.
- Modality Importance: Ablation studies confirm that while audio and visual cues are vital for context (e.g., detecting sarcasm), the textual modality remains the most significant contributor to overall performance.
- Efficiency: GraphSmile maintains a lower training overhead per epoch on IEMOCAP compared to several baseline models, balancing complexity with computational efficiency.
Conclusion and Future Work
GraphSmile successfully addresses the limitations of current MERC models through structured graph fusion and dynamic sentiment tracking. While the model shows high accuracy, it still faces challenges in distinguishing between highly similar emotions (e.g., "Happy" vs. "Excited") and handling class imbalances in datasets like MELD. Future research will explore the integration of Large Language Models (LLMs) to further enhance the framework's reasoning capabilities for deep dialogue understanding.
Summary updated