Abstract: This paper describes the UMONS solution for the OMG-Emotion Challenge. We explore a context-dependent architecture where the arousal and valence of an utterance are predicted according to its surrounding context (i.e. the preceding and following utterances of the video). We report an improvement when taking into account context for both unimodal and multimodal predictions.
Introduction
In affective computing, understanding a single video snippet is often insufficient for accurate emotion recognition. Human emotion is inherently narrative; a joke told early in a video influences the perception of laughter seen much later. The One-Minute Gradual-Emotional Behavior (OMG-Emotion) dataset emphasizes this by requiring labels based on the entire video sequence. This paper presents a solution that moves away from traditional sequential modeling, such as LSTMs, in favor of the Transformer architecture to capture vital temporal context.
Methodology
The proposed pipeline consists of three distinct stages designed to transition from isolated features to a holistic multimodal representation.
1. Context-independent Unimodal Stage
Features are first extracted from each utterance in isolation:
- Text: A CNN with multiple kernel sizes (2, 3, and 4) operating on word2vec embeddings.
- Visual: A 3D-CNN processing 32-frame "cubes" to capture simultaneous motion and spatial features.
- Audio: 121 features extracted via the OpenSmile toolkit, selected through statistical regression from a larger set.
2. Context-dependent Unimodal Stage
Isolated features are fed into a Transformer encoder. The self-attention mechanism allows the model to relate different positions within a video sequence, refining the representation of the current utterance by "looking" at surrounding context.
3. Context-dependent Multimodal Stage
Refined features from all three modalities are fused. The authors utilize Multimodal Compact Bilinear Pooling (CBP), which enables complex interactions between modalities in a high-dimensional space without the parameter explosion typical of standard outer products.
Experimental Setup
The study utilizes the OMG-Emotion dataset, comprising YouTube videos selected for long-term emotional behaviors like monologues and dialogues. The dataset includes 231 training videos (2,442 utterances) and 60 validation videos (617 utterances). Models are evaluated on continuous values for Arousal (intensity) and Valence (positivity/negativity) using the Concordance Correlation Coefficient (CCC).
Results and Discussion
The results demonstrate that incorporating context significantly enhances performance across all modalities. For instance, the visual modality's performance improved from a Mean CCC of 0.186 to 0.227 upon adding the Transformer layer.
The "Contextual Multimodal" approach using CBP fusion achieved the highest performance (Mean CCC of 0.301). This suggests that emotions are best captured through the sophisticated interaction of voice, facial expression, and language, rather than a simple summation of parts. The Transformer's self-attention mechanism provides a more flexible way to weigh relevant video segments compared to the rigid sequential memory of traditional recurrent networks.
Conclusion
While these preliminary results for the OMG-Emotion Challenge are promising, further research is needed to directly compare Transformer performance against Bi-LSTMs on identical feature sets. Additionally, exploring deeper architectures or pre-training on larger datasets could help overcome the limitations of the relatively small OMG-Emotion dataset.
Summary updated