A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition

Ziwang Fu, Feng Liu, Hanyang Wang, Jiayin Qi, Xiangling Fu, Aimin Zhou, Zhibin LiarXiv 2021 · arXiv

Abstract

Multimodal emotion recognition (MER) aims to identify human emotional states by integrating diverse data streams, such as audio and video. While multimodal approaches typically outperform unimodal ones, they often struggle with feature redundancy and the potential loss of original semantic information during fusion. This paper presents CFN-SR, a novel Cross-modal Fusion Network that employs an intra-modal self-attention mechanism and an inter-modal residual structure. By selectively filtering audio features and protecting the structural integrity of video data, CFN-SR achieves a state-of-the-art accuracy of 75.76% on the RAVDESS benchmark with minimal computational overhead.

Introduction

Recognizing human emotions is a complex task involving the interpretation of multiple modalities, including vocal tone (audio) and facial expressions (video). Existing fusion strategies often face three primary hurdles:

  1. Redundancy: Simple concatenation can introduce duplicate information that hinders model performance.
  2. Information Loss: The interaction between modalities can sometimes corrupt the original semantic meaning of individual streams.
  3. Ignoring Complementarity: Models may fail to capture how one modality can compensate for gaps in another.

To address these issues, the authors propose the Cross-modal Fusion Network based on Self-attention and Residual structure (CFN-SR), designed to ensure that the fusion process is additive rather than destructive.

Methodology

The CFN-SR architecture consists of three main stages: feature extraction, cross-modal interaction, and classification.

The CFN-SR architecture and detailed Cross-modal Block.

Feature Extraction

The model utilizes specialized encoders for each modality:

  • Video Stream: A 3D ResNeXt50 captures spatio-temporal features, tracking facial changes over time.
  • Audio Stream: A 1D CNN processes Mel Frequency Cepstrum Coefficient (MFCC) features, representing the "texture" of speech.

The Cross-modal Block

The core innovation lies in the Cross-modal Block, which facilitates intelligent interaction:

  • Intra-modal Self-Attention: Audio features pass through a self-attention mechanism to filter out noise and redundancy, selecting only the most emotionally relevant components.
  • Inter-modal Interaction: These refined audio features are mapped to the video features via linear projection and tanh activation.
  • Residual Structure: A residual connection adds the original video features back into the fused result. This "anchor" ensures the structural integrity of the video stream is maintained during fusion.

Classification

The final fused representation is concatenated with the original modality features and passed through a dense layer for emotion prediction.

Experimental Setup

The model was evaluated on the RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) dataset, containing 1,440 video clips of 24 professional actors expressing eight emotional categories (e.g., neutral, happy, angry). Performance was validated using 5-fold cross-validation to ensure robustness.

Results and Discussion

CFN-SR demonstrates superior performance compared to existing state-of-the-art methods.

Comparison of CFN-SR with state-of-the-art multimodal fusion methods.

Performance Analysis

  • State-of-the-Art Accuracy: CFN-SR achieved 75.76% accuracy, outperforming models like MSAF (74.86%) and MMTM (73.12%).
  • Multimodal Synergy: The model significantly outperformed unimodal baselines (62.99% for video and 56.53% for audio), confirming the effectiveness of the fusion strategy.
  • Efficiency: The Cross-modal Block adds only approximately 30K parameters, representing a highly efficient enhancement to the base architecture.

Key Insights

  • Audio as a Selector: Applying self-attention to audio before fusion acts as a "feature selection" step, making the interaction with video data more efficient.
  • Video as the Anchor: The success of the residual structure on the video modality suggests that video provides a stable spatio-temporal "skeleton" that should be preserved.
  • Directionality: Integrating audio into the video stream (A->V) was found to be more effective than the reverse, likely because the rich structure of video serves as a better foundation for final decision-making.

Conclusion

CFN-SR provides a robust framework for multimodal emotion recognition by balancing feature interaction with information preservation. While currently focused on audio and video, future work will explore the integration of additional modalities, such as text and body gestures, to create a more holistic emotion recognition system.

Summary updated

Abstract

The audio-video based multimodal emotion recognition has attracted a lot of attention due to its robust performance. Most of the existing methods focus on proposing different cross-modal fusion strategies. However, these strategies introduce redundancy in the features of different modalities without fully considering the complementary properties between modal information, and these approaches do not guarantee the non-loss of original semantic information during intra- and inter-modal interactions. In this paper, we propose a novel cross-modal fusion network based on self-attention and residual structure (CFN-SR) for multimodal emotion recognition. Firstly, we perform representation learning for audio and video modalities to obtain the semantic features of the two modalities by efficient ResNeXt and 1D CNN, respectively. Secondly, we feed the features of the two modalities into the cross-modal blocks separately to ensure efficient complementarity and completeness of information through the self-attention mechanism and residual structure. Finally, we obtain the output of emotions by splicing the obtained fused representation with the original representation. To verify the effectiveness of the proposed method, we conduct experiments on the RAVDESS dataset. The experimental results show that the proposed CFN-SR achieves the state-of-the-art and obtains 75.76% accuracy with 26.30M parameters. Our code is available at https://github.com/skeletonNN/CFN-SR.
BibTeX
@misc{fu2021cross,
  author = {Ziwang Fu and Feng Liu and Hanyang Wang and Jiayin Qi and Xiangling Fu and Aimin Zhou and Zhibin Li},
  title = {{A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition}},
  year = {2021},
  eprint = {2111.02172},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.02172},
}
Figure