CIME: Contextual Interaction-Based Multimodal Emotion Analysis With Enhanced Semantic Information
Lists
Only you see this
Abstract
Multimodal emotion analysis is pivotal in decoding complex human affect by integrating diverse data sources such as text, audio, and visual signals. In this article, we introduce contextual interaction-based multimodal emotion analysis with enhanced semantic information (CIME), a novel spatio-temporal interaction network that significantly improves emotion recognition accuracy and robustness. CIME employs a text-centric cross-modal attention mechanism to refine semantic representations, while simultaneously leveraging a graph convolutional network to model contextual dialog information by capturing both intraspeaker and interspeaker relationships. This dual approach enables the effective fusion of modality-specific cues and the mining of latent emotional associations across modalities. Extensive experiments conducted on benchmark datasets—including IEMOCAP and MOSEI—demonstrate that CIME consistently outperforms existing state-of-the-art methods in terms of overall classification accuracy and weighted F1-scores. Furthermore, detailed ablation studies underscore the critical contributions of both the cross-modal attention and graph-based contextual modules.
BibTeX
@article{wang2025cime,
author = {Rui Wang and Chaopeng Guo and Mohammad Shabaz and Imad Rida and Erik Cambria and Xianxun Zhu},
title = {{CIME: Contextual Interaction-Based Multimodal Emotion Analysis With Enhanced Semantic Information}},
journal = {IEEE Transactions on Computational Social Systems},
year = {2025},
doi = {10.1109/tcss.2025.3572495},
}Loading related content...