Abstract
Predicting human emotion along continuous dimensions of arousal (intensity) and valence (positivity/negativity) is a complex task due to the multimodal nature of emotional expression and the temporal dynamics of video data. We present a two-stream deep learning framework that integrates acoustic and visual cues. Our approach utilizes a modified VGG-16 for audio processing (ANet) and a combination of SphereFace with a Bidirectional LSTM for video processing (VNet). By jointly training these models using the Concordance Correlation Coefficient (CCC) loss, we achieve significant performance gains over unimodal baselines on the One-Minute Gradual-Emotion (OMG-Emotion) dataset.
Introduction
Traditional emotion recognition often focuses on discrete categories like "happy" or "sad." However, human emotions are more accurately represented by the "Arousal-Valence" model, which captures emotional states on a continuous spectrum. Automating these predictions from "in-the-wild" video snippets presents several hurdles: effectively fusing multimodal data, handling variable input lengths, and capturing temporal dependencies. This work addresses these challenges through a robust multimodal architecture designed for the OMG-Emotion challenge.
Methodology
The proposed architecture consists of two specialized streams for audio and video, which are fused for joint prediction.
Audio Stream (ANet)
The audio signal is preprocessed into spectrograms via Short-Time Fourier Transform (STFT). We employ a VGG-16 backbone, pretrained on ImageNet, as the feature extractor. Because STFT maps consist of two channels (real and imaginary components), the first layer of the VGG-16 was modified to accept a 2-channel input instead of the standard 3-channel RGB.
Video Stream (VNet)
To manage variable video lengths (ranging from 1 to 30 seconds), we implement a sparse sampling strategy: each snippet is divided into $N_V$ segments, and one frame is randomly sampled from each. Faces are detected and aligned using MTCNN, then processed through SphereFace—a network typically used for face recognition—to extract 512-dimensional feature vectors. These features are sequence-modeled using a Bidirectional LSTM (Bi-LSTM), allowing the model to capture temporal context from both past and future frames.
Multimodal Fusion and Joint Training
Features from the penultimate layers of both ANet and VNet are concatenated and passed through a final fully-connected (FC) layer to predict arousal and valence. While unimodal streams are initially trained using Mean Squared Error (MSE), the joint model is optimized using the Concordance Correlation Coefficient (CCC) to better align with the evaluation metrics of the OMG-Emotion challenge.
Experimental Results
The model was evaluated on the OMG-Emotion dataset, which features video snippets with continuous labels for arousal and valence. Results indicate that while both modalities contribute to the final prediction, visual cues are significantly more informative for this dataset.
| Model | Arousal (CCC) | Valence (CCC) | Total |
|---|---|---|---|
| Audio Baseline | 0.15 | 0.21 | 0.36 |
| Our ANet (Audio) | 0.1879 | 0.256 | 0.4439 |
| Video Baseline | 0.12 | 0.23 | 0.35 |
| Our VNet (Video) | 0.2798 | 0.4688 | 0.7486 |
| Joint Training | 0.3036 | 0.4796 | 0.7832 |
The joint model achieved a total CCC of 0.7832, outperforming the best unimodal stream (VNet) and demonstrating the complementary nature of acoustic and visual information.
Discussion and Conclusion
Our findings highlight several key insights. First, sparse sampling is a highly efficient way to capture temporal dynamics without excessive computational cost. Second, transfer learning from face recognition (SphereFace) provides a strong foundation for emotion feature extraction, suggesting that facial identity and expression features are closely related. Finally, joint training with a CCC-specific loss function is crucial for optimizing performance on continuous emotion dimensions. Future work will explore more sophisticated fusion techniques, such as attention mechanisms, to dynamically weigh modalities based on signal quality.
Summary updated