The OMG-Emotion Behavior Dataset

Pablo V. A. Barros, Nikhil Churamani, Egor Lakomkin, Henrique Siqueira, Alexander Sutherland, Stefan Wermter

TL;DR

The OMG-Emotion (One-Minute Gradual-Emotion) dataset introduces a novel multimodal corpus designed to move AI beyond "instantaneous" emotion recognition toward understanding how emotions evolve over time. By providing long-form YouTube monologues with sequential, context-aware annotations, the researchers offer a challenging new benchmark for training models that perceive emotional transitions as fluidly as humans do.

What Problem Does This Paper Address?

Most current affective computing systems treat emotion recognition like image classification: they look at a short, two-second clip and try to label it as "Happy" or "Sad." However, human emotional expression is rarely that discrete. Our feelings are fluid, contextual, and evolve over time.

The authors identify a significant gap in existing research: a lack of "in-the-wild" data that captures gradual emotion transitions. Existing datasets are either recorded in highly controlled lab settings (like IEMOCAP) or focus on "instantaneous" snapshots that ignore the history of the interaction. Without context—knowing what happened thirty seconds ago—an AI cannot truly understand complex behaviors like sarcasm, empathy, or building frustration.

Methods

The researchers developed a pipeline to create a dataset that prioritizes context:

  1. Data Acquisition: They used a crawler to find YouTube monologues and acting auditions. Monologues were chosen specifically because they allow a single person’s emotional state to evolve naturally within a consistent context.
  2. Processing: They used the OpenSMILE toolkit for voice activity detection and Dlib for face tracking. To ensure data quality, they filtered out clips with multiple speakers or empty transcripts (using Google ASR).
  3. Contextual Annotation: This is the paper's most novel contribution. Using their custom "KT Annotation Tool," they tasked Amazon Mechanical Turk workers with labeling the clips. Crucially, annotators watched the clips in sequence. They were provided with "Context Clips" of everything that happened previously in the video, ensuring their labels for the current utterance were informed by the video's history.
  4. Gold Standard Calculation: Since emotion is subjective, the authors used an Evaluator Weighted Estimator (EWE). Instead of a simple average, EWE weights each annotator's input based on how well they generally agree with the "average" of the group, creating a more robust ground truth for Arousal and Valence.

Datasets

The paper introduces the OMG-Emotion Dataset:

  • Size: 567 videos (approx. 15 hours of footage).
  • Structure: 7,371 individual utterances (average 12.96 per video).
  • Content: "In-the-wild" YouTube monologues.
  • Labels: Both categorical (the six universal Ekman emotions + Neutral) and dimensional (Arousal and Valence scales).

The authors compare OMG-Emotion to existing benchmarks like IEMOCAP (which is scripted/indoor), MOSI (focused on sentiment/gestures), and EmoReact (focused on children). OMG-Emotion stands out by combining "wild" settings, multimodal data (audio, video, text), and a strict contextual annotation strategy.

Key Insights and Ideas

  • The "Monologue" Advantage: By focusing on monologues rather than dialogues, the dataset captures the internal emotional arc of a single individual without the "noise" of turn-taking or overlapping speech.
  • Context is King: The requirement for annotators to watch clips in sequence mimics human perception. We don't judge a person's mood in a vacuum; we judge it based on what they just said.
  • The Challenge of the "Wild": The data shows that "neutral" is the most common state in the wild, and certain emotions like "Fear" or "Surprise" are much rarer than in lab-recorded datasets. This makes the dataset a much more realistic (and difficult) test for AI.

Key Results

The authors provided baseline results using standard architectures (SVMs for audio, CNNs for vision and text) to show the dataset's difficulty:

Modality F1-Score (Categorical) Arousal (CCC) Valence (CCC)
Text (RF T) 0.39 0.15 0.04
Vision (Face) 0.37 0.12 0.23
Audio (SVM) 0.33 0.15 0.21

Note: CCC (Concordance Correlation Coefficient) measures the agreement between the model and the gold standard. A value of 1.0 is perfect agreement.

The relatively low F1-scores and CCC values indicate that OMG-Emotion is a "hard" dataset. No single modality (vision, audio, or text) is currently sufficient to "solve" the dataset, leaving significant room for researchers to develop better multimodal and recurrent neural models.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Data Imbalance: Like many real-world datasets, there is a heavy bias toward "Neutral" and "Happiness," while "Fear," "Disgust," and "Surprise" are underrepresented.
  • Sample Size: While 15 hours is a solid start, the authors aim to increase the number of samples to improve the robustness of deep learning models.
  • Model Complexity: The baselines provided are unimodal. The authors suggest that the real breakthrough will come from models that can effectively fuse audio, video, and text while maintaining a "memory" of the temporal context.

Summary updated

Abstract

This paper is the basis paper for the accepted IJCNN challenge One-Minute Gradual-Emotion Recognition (OMG-Emotion)1 by which we hope to foster long-emotion classification using neural models for the benefit of the IJCNN community. The proposed corpus has as novelty the data collection and annotation strategy based on emotion expressions which evolve over time into a specific context. Different from other corpora, we propose a novel multimodal corpus for emotion expression recognition, which uses gradual annotations with a focus on contextual emotion expressions. Our dataset was collected from Youtube videos using a specific search strategy based on restricted keywords and filtering which guaranteed that the data follow a gradual emotion expression transition, i.e. emotion expressions evolve over time in a natural and continuous fashion. We also provide an experimental protocol and a series of unimodal baseline experiments which can be used to evaluate deep and recurrent neural models in a fair and standard manner.
BibTeX
@inproceedings{barros2018omg,
  author = {Pablo V. A. Barros and Nikhil Churamani and Egor Lakomkin and Henrique Siqueira and Alexander Sutherland and Stefan Wermter},
  title = {{The OMG-Emotion Behavior Dataset}},
  booktitle = {IJCNN},
  year = {2018},
  url = {https://openreview.net/forum?id=Ml66c4SJLJ},
}

Loading related content...

Figure