Abstract
With the advancement of artificial intelligence and computer vision technologies, multimodal emotion recognition has become a prominent research topic. However, existing methods face challenges such as heterogeneous data fusion and the effective utilization of modality correlations. This paper proposes a novel multimodal emotion recognition approach, DeepMSI-MER, based on the integration of contrastive learning and visual sequence compression. The proposed method enhances cross-modal feature fusion through contrastive learning and reduces redundancy in the visual modality by leveraging visual sequence compression. Experimental results on two public datasets, IEMOCAP and MELD, demonstrate that DeepMSI-MER significantly improves the accuracy and robustness of emotion recognition, validating the effectiveness of multimodal feature fusion and the proposed approach.
Introduction
Multimodal Emotion Recognition (MER) aims to understand human emotional states by analyzing three primary data streams: text (what is said), audio (how it is said), and video (facial expressions and body language). Despite its potential, MER faces two significant hurdles. First, heterogeneous data fusion is difficult because text, audio, and video are fundamentally different data types; simple concatenation often fails to capture subtle cross-modal correlations. Second, visual redundancy introduces significant noise; in typical video clips, only a fraction of frames contain the micro-expressions relevant to emotion, while the rest consist of redundant temporal information or irrelevant background pixels.
The DeepMSI-MER Framework
To address these challenges, the authors propose DeepMSI-MER, a framework that utilizes semantic information from audio and text to guide the compression of visual data.
Modality-Specific Feature Extraction
The architecture begins with specialized encoders for each modality. The model utilizes BERT for text and Wav2Vec for audio to extract high-level semantic features. For the visual modality, a Swin Transformer is paired with Temporal Convolutional Networks (TCN) to capture both spatial details and temporal patterns across frames. Research found that sampling exactly 15 frames per video provides the optimal balance for capturing emotional transitions without overloading the TCN.
Visual Sequence Compression (VSC-MSI)
The core innovation of DeepMSI-MER is the Visual Sequence Compression (VSC) module. Rather than processing video in isolation, the model fuses audio and text features into a unified "high-level semantic feature." This feature serves as a filter: similarity scores are calculated between the semantic "vibe" of the audio/text and the visual sequences. Visual data that does not align with this semantic context is treated as noise and pruned, ensuring the model focuses only on emotionally relevant visual information.
Contrastive Learning and Late Fusion
In the final stage, the model employs a contrastive learning loss function during late fusion. This objective forces the model to maximize the similarity between features sharing the same emotional label (positive samples) while increasing the distance between features from different emotional categories (negative samples). This process refines the model's ability to distinguish between nuanced emotions, such as "Angry" versus "Frustrated." Furthermore, the contrastive loss gradient derivation reveals an automatic "gradient balancing" effect, allowing the model to focus its learning capacity on harder-to-distinguish samples.
Experimental Evaluation
The researchers evaluated DeepMSI-MER on two benchmark datasets: IEMOCAP (scripted and improvised actor performances) and MELD (in-the-wild utterances from the TV show Friends).
Key Results
DeepMSI-MER demonstrated superior performance over modern baselines like DialogueGCN and GS-MCC:
- IEMOCAP: Achieved an average accuracy and F1-score of 84.7%, with particularly high performance in the "Sad" (87.5%) and "Angry" (89.4%) categories.
- MELD: Achieved an average accuracy of 69.4%. While the chaotic nature of sitcom data presents more challenges, the model showed significant gains in "Neutral" and "Surprise" categories.
- Ablation Studies: Removing any single modality (Text, Audio, or Video) resulted in a sharp performance decline, confirming that the fusion strategy effectively leverages the complementary strengths of each data source.
Conclusion and Future Work
DeepMSI-MER effectively addresses visual redundancy and cross-modal fusion through its novel compression and contrastive learning approach. However, challenges remain in distinguishing between closely related emotions like "Neutral" and "Sadness" in unconstrained environments. Future work will focus on refining semantic similarity calculations and addressing class imbalances for rare emotions like "Fear" and "Disgust."
Summary updated