Muse-ing on the Impact of Utterance Ordering on Crowdsourced Emotion Annotations

Mimansa Jaiswal, Zakaria Aldeneh, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, Emily Mower Provost

Introduction

In the field of Emotion AI, establishing a reliable "ground truth" is a complex challenge. Because emotion is inherently subjective, researchers typically rely on crowdsourced workers to provide labels for audio or video clips. However, a significant disconnect exists between how these labels are generated and how they are utilized in machine learning. While human annotators often view clips in their original chronological sequence—allowing them to follow a speaker's narrative and establish a baseline—the resulting machine learning models are frequently "static," analyzing single utterances in isolation.

This study investigates whether the presentation order of audio clips (randomized vs. sequential) fundamentally alters the labels provided by humans and, consequently, how those labels impact the performance of static and dynamic AI models.

The Multimodal Stressed Emotion (MuSE) Dataset

To explore these questions, the researchers introduced the Multimodal Stressed Emotion (MuSE) dataset. This dataset consists of 2,648 utterances (~10 hours of audio) from 28 participants. Recordings were captured under two distinct conditions: "Stressed" (during final exam periods) and "Non-Stressed."

Participants responded to monologue prompts designed to elicit emotional responses, such as describing significant life accomplishments. Crucially, after each section, speakers provided self-reported emotion ratings. This self-report serves as a rare and valuable baseline for "true" internal state, allowing the researchers to compare crowdsourced labels against the speaker's own perspective.

Experimental Methodology

The study employed a controlled experiment with two crowdsourcing conditions:

  1. Randomized: Annotators were shown clips in a completely random order, stripped of chronological context.
  2. Contextualized: Annotators viewed clips in their original sequence, allowing them to follow the speaker's story.

To evaluate the impact of these labels on AI, the researchers developed two model architectures:

  • Static (DNN): A standard Deep Neural Network that predicts emotion based on a single utterance.
  • Dynamic (GRU): A Gated Recurrent Unit network designed to "remember" previous utterances, mimicking the chronological context available to the sequential annotators.

The analysis focused on two primary emotional dimensions: Activation (the intensity of the emotion) and Valence (the positivity or negativity of the emotion).

Key Insights

1. Context Improves Accuracy relative to the Speaker

The study found that annotators who viewed clips in sequence produced labels that were significantly closer to the speaker's self-reported emotions ($p < 0.01$). This suggests that context allows "outsiders" to better perceive the internal state of the speaker, rather than relying solely on external acoustic cues.

2. Randomization Increases Consensus

Interestingly, inter-rater agreement was higher in the randomized condition. Without the nuance provided by context, annotators tended to rely on stereotypical acoustic markers (e.g., assuming high volume always indicates high activation). This leads to more uniform labels across different workers, but those labels may be less accurate representations of the speaker's actual feelings.

3. The "Context Mismatch" Penalty

A critical finding of the research is the penalty incurred when there is a mismatch between the data labeling process and the model architecture. When a static model (which lacks context) was trained on contextual labels, its performance on Valence dropped. The model struggled because the human labels were influenced by information (the preceding clips) that the model could not access.

4. Contextual Labels Require Contextual Models

The benefits of contextual labeling are best realized when paired with dynamic models. When using a GRU with four utterances of history, the performance gain for Activation was 10.5% for contextual labels, compared to only 4.41% for randomized labels. This indicates that if researchers provide context to annotators, they must also provide it to the model to capture the full emotional signal.

Conclusion

The research highlights that the way we present data to human annotators is not just a matter of convenience; it fundamentally changes the nature of the resulting dataset. While sequential presentation helps annotators align with a speaker's self-report, it necessitates the use of dynamic models that can account for that same context. For developers of Emotion AI, this underscores the importance of aligning the "human-in-the-loop" labeling process with the eventual model architecture to avoid the pitfalls of contextual mismatch.

Summary updated

Abstract

Emotion recognition algorithms rely on data annotated with high quality labels. However, emotion expression and perception are inherently subjective. There is generally not a single annotation that can be unambiguously declared "correct". As a result, annotations are colored by the manner in which they were collected. In this paper, we conduct crowdsourcing experiments to investigate this impact on both the annotations themselves and on the performance of these algorithms. We focus on one critical question: the effect of context. We present a new emotion dataset, Multimodal Stressed Emotion (MuSE), and annotate the dataset using two conditions: randomized, in which annotators are presented with clips in random order, and contextualized, in which annotators are presented with clips in order. We find that contextual labeling schemes result in annotations that are more similar to a speaker's own self-reported labels and that labels generated from randomized schemes are most easily predictable by automated systems.
BibTeX
@inproceedings{jaiswal2019muse,
  author = {Mimansa Jaiswal and Zakaria Aldeneh and Cristian-Paul Bara and Yuanhang Luo and Mihai Burzo and Rada Mihalcea and Emily Mower Provost},
  title = {{Muse-ing on the Impact of Utterance Ordering on Crowdsourced Emotion Annotations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8682793},
}
Figure