A Deep Network for Arousal-Valence Emotion Prediction with Acoustic-Visual Cues

Songyou Peng, Le Zhang, Yutong Ban, Meng Fang, Stefan WinklerarXiv 2018 · OpenReview

Abstract

Predicting human emotion along continuous dimensions of arousal (intensity) and valence (positivity/negativity) is a complex task due to the multimodal nature of emotional expression and the temporal dynamics of video data. We present a two-stream deep learning framework that integrates acoustic and visual cues. Our approach utilizes a modified VGG-16 for audio processing (ANet) and a combination of SphereFace with a Bidirectional LSTM for video processing (VNet). By jointly training these models using the Concordance Correlation Coefficient (CCC) loss, we achieve significant performance gains over unimodal baselines on the One-Minute Gradual-Emotion (OMG-Emotion) dataset.

Introduction

Traditional emotion recognition often focuses on discrete categories like "happy" or "sad." However, human emotions are more accurately represented by the "Arousal-Valence" model, which captures emotional states on a continuous spectrum. Automating these predictions from "in-the-wild" video snippets presents several hurdles: effectively fusing multimodal data, handling variable input lengths, and capturing temporal dependencies. This work addresses these challenges through a robust multimodal architecture designed for the OMG-Emotion challenge.

Methodology

The proposed architecture consists of two specialized streams for audio and video, which are fused for joint prediction.

Architecture of the proposed multimodal emotion prediction network, showing the ANet (audio) and VNet (video) streams.

Audio Stream (ANet)

The audio signal is preprocessed into spectrograms via Short-Time Fourier Transform (STFT). We employ a VGG-16 backbone, pretrained on ImageNet, as the feature extractor. Because STFT maps consist of two channels (real and imaginary components), the first layer of the VGG-16 was modified to accept a 2-channel input instead of the standard 3-channel RGB.

Video Stream (VNet)

To manage variable video lengths (ranging from 1 to 30 seconds), we implement a sparse sampling strategy: each snippet is divided into $N_V$ segments, and one frame is randomly sampled from each. Faces are detected and aligned using MTCNN, then processed through SphereFace—a network typically used for face recognition—to extract 512-dimensional feature vectors. These features are sequence-modeled using a Bidirectional LSTM (Bi-LSTM), allowing the model to capture temporal context from both past and future frames.

Multimodal Fusion and Joint Training

Features from the penultimate layers of both ANet and VNet are concatenated and passed through a final fully-connected (FC) layer to predict arousal and valence. While unimodal streams are initially trained using Mean Squared Error (MSE), the joint model is optimized using the Concordance Correlation Coefficient (CCC) to better align with the evaluation metrics of the OMG-Emotion challenge.

Experimental Results

The model was evaluated on the OMG-Emotion dataset, which features video snippets with continuous labels for arousal and valence. Results indicate that while both modalities contribute to the final prediction, visual cues are significantly more informative for this dataset.

Model Arousal (CCC) Valence (CCC) Total
Audio Baseline 0.15 0.21 0.36
Our ANet (Audio) 0.1879 0.256 0.4439
Video Baseline 0.12 0.23 0.35
Our VNet (Video) 0.2798 0.4688 0.7486
Joint Training 0.3036 0.4796 0.7832
Final performance metrics for the joint training approach.

The joint model achieved a total CCC of 0.7832, outperforming the best unimodal stream (VNet) and demonstrating the complementary nature of acoustic and visual information.

Discussion and Conclusion

Our findings highlight several key insights. First, sparse sampling is a highly efficient way to capture temporal dynamics without excessive computational cost. Second, transfer learning from face recognition (SphereFace) provides a strong foundation for emotion feature extraction, suggesting that facial identity and expression features are closely related. Finally, joint training with a CCC-specific loss function is crucial for optimizing performance on continuous emotion dimensions. Future work will explore more sophisticated fusion techniques, such as attention mechanisms, to dynamically weigh modalities based on signal quality.

Summary updated

Abstract

In this paper, we comprehensively describe the methodology of our submissions to the One-Minute Gradual-Emotion Behavior Challenge 2018.
BibTeX
@misc{peng2018deep,
  author = {Songyou Peng and Le Zhang and Yutong Ban and Meng Fang and Stefan Winkler},
  title = {{A Deep Network for Arousal-Valence Emotion Prediction with Acoustic-Visual Cues}},
  year = {2018},
  eprint = {1805.00638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.00638},
}
Figure