Analyzing the Influence of Dataset Composition for Emotion Recognition

Alexander Sutherland, Sven Magg, Cornelius Weber, Stefan WermterarXiv 2021 · arXiv

Abstract

Multimodal emotion recognition models often report high accuracy on benchmark datasets like IEMOCAP. However, this paper demonstrates that much of this performance stems from dataset composition issues rather than robust learning. By analyzing the scripted nature of IEMOCAP versus the naturalistic OMG-Emotion Behavior dataset, the authors show that overlapping sentences lead to severe overfitting. Models trained on scripted data fail to generalize to real-world scenarios, posing a significant challenge for affective computing and Human-Robot Interaction (HRI).

Introduction

In the field of affective computing, combining audio, video, and text is the standard approach for emotion recognition. Recently, textual features have shown remarkable performance, sometimes outperforming other modalities. This paper investigates whether this success reflects a breakthrough in natural language understanding or is merely an artifact of how benchmark datasets are constructed. Specifically, the authors examine the IEMOCAP dataset, suspecting that its scripted, repetitive dialogue allows models to "cheat" by memorizing specific phrases rather than learning emotional context.

Methodology

The authors developed a 1D Convolutional Neural Network (1D-CNN) to process textual data at two levels:

  • Word Embeddings: Utilizing pre-trained Google News Word Embeddings.
  • Semantic Embeddings: Using SEMAFOR frame-semantic parsers.

The architecture employs separate channels for each input type, followed by temporal convolution and global max pooling. A late fusion strategy concatenates these features before a final fully connected layer performs classification. To evaluate generalization, the researchers conducted both 10-fold cross-validation and cross-dataset testing between IEMOCAP and the OMG-Emotion Behavior dataset.

Dataset Comparison

The study utilizes two primary datasets:

  1. IEMOCAP: Contains 5,616 utterances from actors in scripted and improvised scenarios.
  2. OMG Emotion-Behavior Dataset: Comprises 4,656 crowd-sourced YouTube videos, providing a more naturalistic and unscripted baseline.

Both datasets were mapped to four emotion classes: Angry, Happy/Excited, Neutral, and Sad.

The Impact of Scripted Data

A core finding is the "Illusion of High Accuracy" on IEMOCAP. Because the scripted sessions involve actors reading identical lines, there is massive data overlap between training and test sets.

Overlapping data proportionality across different dataset subsets as a function of utterance length.

As shown in Figure 1, the proportion of overlapping data is significantly higher in scripted IEMOCAP compared to improvised subsets or the OMG dataset. This overlap allows models to achieve high accuracy through simple memorization.

Furthermore, this memorization leads to extreme overconfidence. When classifying scripted utterances, the model's softmax activations frequently cluster between 0.95 and 1.0.

Distribution of softmax activations showing model overconfidence on scripted data.

This abnormal certainty, visualized in Figure 2, is a classic symptom of overfitting. The model is not learning generalizable emotional features but is instead latching onto dataset-specific keywords—a phenomenon known as the "Clever Hans" effect.

Experimental Results

The results highlight a stark contrast between in-domain performance and real-world generalization:

  • In-Domain Performance: The fusion model achieved 67.7% accuracy on IEMOCAP but only 43.5% on the naturalistic OMG dataset.
  • Cross-Dataset Generalization: Performance dropped significantly when testing across datasets. A model trained on IEMOCAP achieved only 35.2% accuracy on OMG.
  • The Value of Improvised Data: Interestingly, training on the smaller improvised subset of IEMOCAP yielded better generalization to OMG (35.6%) than training on the full dataset, confirming that scripted data hinders the learning of robust features.

Conclusion

The authors conclude that current training procedures on scripted datasets like IEMOCAP produce unreliable models for real-world application. They urge the research community to prioritize unscripted, naturalistic datasets and to account for data overlap during model evaluation. Ensuring that emotional recognition systems can generalize beyond specific scripts is essential for the development of reliable HRI systems.

Summary updated

Abstract

Recognizing emotions from text in multimodal architectures has yielded promising results, surpassing video and audio modalities under certain circumstances. However, the method by which multimodal data is collected can be significant for recognizing emotional features in language. In this paper, we address the influence data collection methodology has on two multimodal emotion recognition datasets, the IEMOCAP dataset and the OMG-Emotion Behavior dataset, by analyzing textual dataset compositions and emotion recognition accuracy. Experiments with the full IEMOCAP dataset indicate that the composition negatively influences generalization performance when compared to the OMG-Emotion Behavior dataset. We conclude by discussing the impact this may have on HRI experiments.
BibTeX
@misc{sutherland2021analyzing,
  author = {Alexander Sutherland and Sven Magg and Cornelius Weber and Stefan Wermter},
  title = {{Analyzing the Influence of Dataset Composition for Emotion Recognition}},
  year = {2021},
  eprint = {2103.03700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.03700},
}
Figure