Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition

Tong Shi, Xuri Ge, Joemon M. Jose, Nicolas Pugeault, Paul Henderson

Abstract

Audio-Visual Emotion Recognition (AVER) is a critical component of affective computing, yet it remains challenging due to the subtle nature of facial expressions and the complexity of cross-modal fusion. This paper introduces DE-III, a novel framework designed to enhance emotion recognition by explicitly capturing fine-grained facial "micro-expressions" through optical flow and integrating them with auditory signals via a sophisticated attention mechanism. By utilizing Pair-wise Attention Enhancement (PAE) and Inter-modal Feature Enhancement (IFE), the DE-III model achieves state-of-the-art performance across three major benchmarks: CREMA-D, RAVDESS, and MSP-IMPROV.

Introduction

Human emotional expression is inherently multi-modal, combining vocal inflections with facial movements. While deep learning has advanced the field of Audio-Visual Emotion Recognition (AVER), two primary obstacles persist. First, visual signals often contain subtle, high-frequency changes—such as the twitch of a muscle or a slight furrowing of the brow—that standard RGB-based models frequently overlook. Second, the fusion of audio and video is often handled through simple linear methods or shared networks that fail to account for the unique distributions and non-linear relationships between the two modalities.

To address these issues, researchers from the University of Glasgow developed the Detail-Enhanced Intra- and Inter-modal Interaction (DE-III) network. This framework prioritizes the extraction of fine-grained visual details and employs a robust cross-modal attention strategy to ensure that both audio and visual cues are optimally utilized for emotion classification.

Methodology

The DE-III architecture is built upon several innovative modules designed to capture both local nuances and global context.

1. Visual Detail Enhancement via Optical Flow

To capture "texture variations" and rapid facial motion, the model extracts optical flow between adjacent video frames. Unlike static RGB frames, optical flow highlights the velocity and direction of movement, making it easier for the network to identify micro-expressions. This information is processed in parallel with standard RGB data.

2. Pair-wise Attention Enhancement (PAE)

Rather than simply concatenating RGB and optical flow features, the DE-III model employs a PAE module. This component calculates dynamic weights for each visual stream, allowing the network to prioritize either the raw image or the motion data depending on which is more informative at a specific point in time.

3. Conformer-Based Encoding

The model utilizes Conformer encoders—a hybrid architecture combining Transformers and Convolutions—to process both audio (extracted via wav2vec 2.0) and video features. This allows the model to capture global temporal context while maintaining a focus on local relationships within the data.

4. Inter-modal Feature Enhancement (IFE)

The IFE module facilitates communication between modalities using cross-attention. This allows one modality to "query" the other; for instance, video features can attend to audio fragments to resolve ambiguity in a facial expression. The researchers found that letting video features attend to audio (V-cross) was particularly effective for decoding complex emotional states.

5. Multi-Head Training Strategy

To ensure robust feature learning, the model is trained with three independent prediction heads: Video-centric, Audio-centric, and a final Fusion head. This multi-task approach forces each branch of the network to learn discriminative features before they are combined for final inference.

Experimental Results

The DE-III model was evaluated on three prominent datasets, setting new benchmarks for performance in each case.

CREMA-D

On the CREMA-D dataset, which contains 7,442 clips labeled with six discrete emotions, DE-III achieved an 83.7% F1-Micro score. This represents a 3.4% improvement over the previous state-of-the-art model, LADDER.

RAVDESS

In evaluations using the RAVDESS dataset (2,880 clips of eight emotional states), the model reached 82.7% accuracy, surpassing the previous best of 81.6% held by the MMER framework.

MSP-IMPROV

For continuous emotion recognition (Valence, Arousal, and Dominance), DE-III achieved exceptionally high Concordance Correlation Coefficients (CCC):

  • Valence: 89.3%
  • Arousal: 88.7%
  • Dominance: 85.8%

Discussion and Insights

A key finding of this research is that "motion is a modality." By treating optical flow as a primary source of texture detail, the model compensates for the relative stillness of facial frames compared to the high-frequency nature of audio. Ablation studies further revealed that fine-grained timing is essential; using a "stride of 1" (processing every single frame transition) yielded the best results. Increasing the stride or window length led to a drop in performance, suggesting that emotional cues are often embedded in the highest-frequency visual changes.

Conclusion

DE-III demonstrates the power of combining fine-grained visual motion analysis with sophisticated inter-modal attention. While the model currently requires significant computational resources for training and inference, its success on major benchmarks highlights the importance of capturing subtle facial dynamics in affective computing. Future work will likely focus on optimizing the model for real-time deployment and testing its robustness against "in-the-wild" data characterized by poor lighting and occlusions.

Summary updated

Abstract

Capturing complex temporal relationships between video and audio modalities is vital for Audio-Visual Emotion Recognition (AVER). However, existing methods lack attention to local details, such as facial state changes between video frames, which can reduce the discriminability of features and thus lower recognition accuracy. In this paper, we propose a Detail-Enhanced Intra- and Inter-modal Interaction network (DE-III) for AVER, incorporating several novel aspects. We introduce optical flow information to enrich video representations with texture details that better capture facial state changes. A fusion module integrates the optical flow estimation with the corresponding video frames to enhance the representation of facial texture variations. We also design attentive intra- and inter-modal feature enhancement modules to further improve the richness and discriminability of video and audio representations. A detailed quantitative evaluation shows that our proposed model outperforms all existing methods on three benchmark datasets for both concrete and continuous emotion recognition. To encourage further research and ensure replicability, our project code is public available at https://github.com/stonewalking/DE-III.
BibTeX
@inproceedings{shi2024detail,
  author = {Tong Shi and Xuri Ge and Joemon M. Jose and Nicolas Pugeault and Paul Henderson},
  title = {{Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition}},
  booktitle = {ICPR},
  year = {2024},
  url = {https://openreview.net/forum?id=pKmGSGA3MD},
}
Figure