Abstract
Multimodal emotion recognition (MER) aims to identify human emotional states by integrating diverse data streams, such as audio and video. While multimodal approaches typically outperform unimodal ones, they often struggle with feature redundancy and the potential loss of original semantic information during fusion. This paper presents CFN-SR, a novel Cross-modal Fusion Network that employs an intra-modal self-attention mechanism and an inter-modal residual structure. By selectively filtering audio features and protecting the structural integrity of video data, CFN-SR achieves a state-of-the-art accuracy of 75.76% on the RAVDESS benchmark with minimal computational overhead.
Introduction
Recognizing human emotions is a complex task involving the interpretation of multiple modalities, including vocal tone (audio) and facial expressions (video). Existing fusion strategies often face three primary hurdles:
- Redundancy: Simple concatenation can introduce duplicate information that hinders model performance.
- Information Loss: The interaction between modalities can sometimes corrupt the original semantic meaning of individual streams.
- Ignoring Complementarity: Models may fail to capture how one modality can compensate for gaps in another.
To address these issues, the authors propose the Cross-modal Fusion Network based on Self-attention and Residual structure (CFN-SR), designed to ensure that the fusion process is additive rather than destructive.
Methodology
The CFN-SR architecture consists of three main stages: feature extraction, cross-modal interaction, and classification.
Feature Extraction
The model utilizes specialized encoders for each modality:
- Video Stream: A 3D ResNeXt50 captures spatio-temporal features, tracking facial changes over time.
- Audio Stream: A 1D CNN processes Mel Frequency Cepstrum Coefficient (MFCC) features, representing the "texture" of speech.
The Cross-modal Block
The core innovation lies in the Cross-modal Block, which facilitates intelligent interaction:
- Intra-modal Self-Attention: Audio features pass through a self-attention mechanism to filter out noise and redundancy, selecting only the most emotionally relevant components.
- Inter-modal Interaction: These refined audio features are mapped to the video features via linear projection and
tanhactivation. - Residual Structure: A residual connection adds the original video features back into the fused result. This "anchor" ensures the structural integrity of the video stream is maintained during fusion.
Classification
The final fused representation is concatenated with the original modality features and passed through a dense layer for emotion prediction.
Experimental Setup
The model was evaluated on the RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) dataset, containing 1,440 video clips of 24 professional actors expressing eight emotional categories (e.g., neutral, happy, angry). Performance was validated using 5-fold cross-validation to ensure robustness.
Results and Discussion
CFN-SR demonstrates superior performance compared to existing state-of-the-art methods.
Performance Analysis
- State-of-the-Art Accuracy: CFN-SR achieved 75.76% accuracy, outperforming models like MSAF (74.86%) and MMTM (73.12%).
- Multimodal Synergy: The model significantly outperformed unimodal baselines (62.99% for video and 56.53% for audio), confirming the effectiveness of the fusion strategy.
- Efficiency: The Cross-modal Block adds only approximately 30K parameters, representing a highly efficient enhancement to the base architecture.
Key Insights
- Audio as a Selector: Applying self-attention to audio before fusion acts as a "feature selection" step, making the interaction with video data more efficient.
- Video as the Anchor: The success of the residual structure on the video modality suggests that video provides a stable spatio-temporal "skeleton" that should be preserved.
- Directionality: Integrating audio into the video stream (A->V) was found to be more effective than the reverse, likely because the rich structure of video serves as a better foundation for final decision-making.
Conclusion
CFN-SR provides a robust framework for multimodal emotion recognition by balancing feature interaction with information preservation. While currently focused on audio and video, future work will explore the integration of additional modalities, such as text and body gestures, to create a more holistic emotion recognition system.
Summary updated