Abstract
This paper is the basis paper for the accepted IJCNN challenge One-Minute Gradual-Emotion Recognition (OMG-Emotion) by which we hope to foster long-emotion classification using neural models for the benefit of the IJCNN community. The proposed corpus has as the novelty the data collection and annotation strategy based on emotion expressions which evolve over time into a specific context. Different from other corpora, we propose a novel multimodal corpus for emotion expression recognition, which uses gradual annotations with a focus on contextual emotion expressions. Our dataset was collected from Youtube videos using a specific search strategy based on restricted keywords and filtering which guaranteed that the data follow a gradual emotion expression transition, i.e. emotion expressions evolve over time in a natural and continuous fashion. We also provide an experimental protocol and a series of unimodal baseline experiments which can be used to evaluate deep and recurrent neural models in a fair and standard manner.
Introduction
Most current affective computing systems treat emotion recognition as a discrete classification task, analyzing short, isolated clips to assign labels like "Happy" or "Sad." However, human emotional expression is inherently fluid and contextual. The OMG-Emotion (One-Minute Gradual-Emotion) dataset addresses the gap in "in-the-wild" data capturing gradual emotional transitions. By moving beyond instantaneous snapshots, this dataset enables the training of models that perceive emotional evolution as it occurs in natural human interaction.
Methodology
The researchers developed a robust pipeline to prioritize context in the dataset creation. Data was crawled from YouTube, specifically targeting monologues and acting auditions where a single individual's emotional state evolves naturally. Voice activity detection (OpenSMILE) and face tracking (Dlib) were employed to filter and process the raw footage.
Contextual Annotation
A critical innovation of this work is the contextual annotation strategy. Using the custom "KT Annotation Tool," annotators labeled clips in sequence. Unlike traditional methods where clips are randomized, annotators were provided with "Context Clips" of previous segments to ensure their labels were informed by the video's history.
To handle the subjectivity of emotion, the authors utilized an Evaluator Weighted Estimator (EWE). This method weights annotator input based on their general agreement with the group average, creating a more robust ground truth for dimensional (Arousal/Valence) and categorical labels.
The OMG-Emotion Dataset
The resulting dataset consists of 567 videos totaling approximately 15 hours of footage, divided into 7,371 utterances. It provides both categorical labels (the six universal Ekman emotions plus Neutral) and dimensional labels.
As shown in the distributions, "Neutral" is the most frequent state, reflecting real-world conditions. The dataset presents a significant challenge due to this natural imbalance and the subtlety of emotional transitions compared to scripted, lab-recorded datasets like IEMOCAP.
Experimental Results
Baseline experiments were conducted using unimodal architectures, including Random Forests for text, CNNs for vision, and SVMs for audio.
| Modality | F1-Score (Categorical) | Arousal (CCC) | Valence (CCC) |
|---|---|---|---|
| Text (RF T) | 0.39 | 0.15 | 0.04 |
| Vision (Face) | 0.37 | 0.12 | 0.23 |
| Audio (SVM) | 0.33 | 0.15 | 0.21 |
Note: CCC (Concordance Correlation Coefficient) measures the agreement between the model and the gold standard.
The relatively low F1-scores and CCC values highlight the difficulty of the OMG-Emotion benchmark. No single modality is currently sufficient to achieve high performance, suggesting that future breakthroughs will require sophisticated multimodal fusion and temporal modeling.
Conclusion
The OMG-Emotion dataset provides a novel benchmark for context-aware emotion recognition. By capturing long-form emotional arcs in the wild, it challenges the AI community to move toward more human-like perception. Future work will focus on expanding the dataset size and developing recurrent neural models capable of maintaining a memory of emotional context.
Summary updated