Abstract
Multimodal emotion recognition models often report high accuracy on benchmark datasets like IEMOCAP. However, this paper demonstrates that much of this performance stems from dataset composition issues rather than robust learning. By analyzing the scripted nature of IEMOCAP versus the naturalistic OMG-Emotion Behavior dataset, the authors show that overlapping sentences lead to severe overfitting. Models trained on scripted data fail to generalize to real-world scenarios, posing a significant challenge for affective computing and Human-Robot Interaction (HRI).
Introduction
In the field of affective computing, combining audio, video, and text is the standard approach for emotion recognition. Recently, textual features have shown remarkable performance, sometimes outperforming other modalities. This paper investigates whether this success reflects a breakthrough in natural language understanding or is merely an artifact of how benchmark datasets are constructed. Specifically, the authors examine the IEMOCAP dataset, suspecting that its scripted, repetitive dialogue allows models to "cheat" by memorizing specific phrases rather than learning emotional context.
Methodology
The authors developed a 1D Convolutional Neural Network (1D-CNN) to process textual data at two levels:
- Word Embeddings: Utilizing pre-trained Google News Word Embeddings.
- Semantic Embeddings: Using SEMAFOR frame-semantic parsers.
The architecture employs separate channels for each input type, followed by temporal convolution and global max pooling. A late fusion strategy concatenates these features before a final fully connected layer performs classification. To evaluate generalization, the researchers conducted both 10-fold cross-validation and cross-dataset testing between IEMOCAP and the OMG-Emotion Behavior dataset.
Dataset Comparison
The study utilizes two primary datasets:
- IEMOCAP: Contains 5,616 utterances from actors in scripted and improvised scenarios.
- OMG Emotion-Behavior Dataset: Comprises 4,656 crowd-sourced YouTube videos, providing a more naturalistic and unscripted baseline.
Both datasets were mapped to four emotion classes: Angry, Happy/Excited, Neutral, and Sad.
The Impact of Scripted Data
A core finding is the "Illusion of High Accuracy" on IEMOCAP. Because the scripted sessions involve actors reading identical lines, there is massive data overlap between training and test sets.
As shown in Figure 1, the proportion of overlapping data is significantly higher in scripted IEMOCAP compared to improvised subsets or the OMG dataset. This overlap allows models to achieve high accuracy through simple memorization.
Furthermore, this memorization leads to extreme overconfidence. When classifying scripted utterances, the model's softmax activations frequently cluster between 0.95 and 1.0.
This abnormal certainty, visualized in Figure 2, is a classic symptom of overfitting. The model is not learning generalizable emotional features but is instead latching onto dataset-specific keywords—a phenomenon known as the "Clever Hans" effect.
Experimental Results
The results highlight a stark contrast between in-domain performance and real-world generalization:
- In-Domain Performance: The fusion model achieved 67.7% accuracy on IEMOCAP but only 43.5% on the naturalistic OMG dataset.
- Cross-Dataset Generalization: Performance dropped significantly when testing across datasets. A model trained on IEMOCAP achieved only 35.2% accuracy on OMG.
- The Value of Improvised Data: Interestingly, training on the smaller improvised subset of IEMOCAP yielded better generalization to OMG (35.6%) than training on the full dataset, confirming that scripted data hinders the learning of robust features.
Conclusion
The authors conclude that current training procedures on scripted datasets like IEMOCAP produce unreliable models for real-world application. They urge the research community to prioritize unscripted, naturalistic datasets and to account for data overlap during model evaluation. Ensuring that emotional recognition systems can generalize beyond specific scripts is essential for the development of reliable HRI systems.
Summary updated