A Mapping on Current Classifying Categories of Emotions Used in Multimodal Models for Emotion Recognition

Ziwei Gong, Muyin Yao, Xinyi Hu, Xiaoning Zhu, Julia HirschbergEACL 2024

TL;DR

This paper tackles the challenge of inconsistent emotion categories across multimodal emotion recognition datasets by proposing a novel mapping method grounded in psychological theories. This mapping unifies disparate emotion labels, enabling better model comparison, dataset adaptation, and potentially improving the accuracy of emotion recognition systems.

What Problem Does This Paper Address?

Emotion recognition, a crucial aspect of human-computer interaction, has seen significant advancements with the rise of multimodal approaches (using audio, video, and text). However, a major hurdle is the lack of standardized emotion classification systems across different datasets. Researchers often face a "Tower of Babel" situation where datasets use varying sets of emotion labels, sometimes based on different psychological theories (e.g., Ekman's basic emotions, Plutchik's evolutionary model, or dimensional approaches). This inconsistency makes it difficult to compare models trained on different datasets, combine datasets for training, or even understand how different models are interpreting emotions. The paper aims to bridge this gap by creating a unified mapping that connects these diverse categorical systems.

Methods

The authors propose a psychology-based mapping method to reconcile different emotion classification schemes. Their approach involves:

  • Identifying Common Emotions: Emotions that appear across multiple classification systems are kept as is, assuming annotators can consistently identify them.
  • Mapping Higher-Level Emotions: Emotions unique to certain categories (like "anticipation" or "surprise") are mapped based on existing literature, considering their valence (positivity/negativity) and arousal (intensity) levels. Emotions with similar valence and arousal are considered more closely related.
  • Human Evaluation: When mapping choices are ambiguous or tied, human annotators are employed to determine the most accurate mapping. This was particularly crucial for emotions like "surprise," where its valence can be context-dependent.

The paper also details an "Annotation Process" where annotators are provided with clips, scripts, and emotion definitions to ensure consistency, with at least two annotators per data point.

Datasets

The study utilizes several publicly available datasets to develop and validate their mapping method:

  • MEmoR: A multimodal dataset for emotion recognition in conversations.
  • MELD: Another multimodal dataset for emotion recognition in conversations.
  • IEMOCAP: A multimodal dataset focusing on facial expressions and speech.
  • FER-2013: A computer vision dataset primarily for facial emotion recognition.

These datasets represent a range of emotion classification schemes, from 14 fine-grained emotions to 3 sentiment categories. The paper reconstructs these datasets based on their proposed mapping to evaluate its impact on machine learning models.

Key Insights and Ideas

  • The "Tower of Babel" in Emotion Recognition: The core problem is the lack of a common language for emotions across datasets, hindering progress.
  • Psychology as a Bridge: Leveraging established psychological theories provides a robust foundation for creating a unified emotion mapping.
  • Valence and Arousal as Key Dimensions: These two psychological constructs are crucial for understanding the relationships between different emotions, especially for higher-level or ambiguous ones.
  • Human Judgment is Essential: For complex or context-dependent emotions, human evaluation is necessary to ensure accurate mapping.
  • Fewer Categories, Higher Accuracy (Generally): The experiments suggest that models tend to perform better with fewer, broader emotion categories, simplifying the classification task.

Key Results

The paper demonstrates that their mapping method achieves high accuracy in re-annotating datasets, with inter-annotator agreement scores indicating its reliability.

  • Mapping Accuracy: Re-annotation achieved an accuracy of 0.96 (Annotator 1) and 0.917 (Annotator 2) with fair inter-annotator agreement (0.318 Cohen's Kappa).
  • Model Performance: When applied to machine learning models (CNN and MEmoR), the mapping generally led to improved performance when reducing the number of emotion categories. For instance, a CNN model showed higher accuracy with 6 categories compared to 7, and a significant jump from 7 to 3 categories.
  • Surprise to Anger Mapping: Human evaluations favored mapping "surprise" to "anger," likely due to annotators associating surprise with negative valence and high arousal, aligning with anger's characteristics.
  • Negative Emotions are More Fine-Grained: Analysis of the mapping revealed that negative emotions are more granularly represented across different classification systems compared to positive or neutral emotions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope of Mapping: The proposed mapping is limited to a specific range of 3 to 14 categories. For highly specialized tasks requiring finer distinctions, this mapping might need expansion.
  • Dataset Dependency: The accuracy of the mapping is influenced by the characteristics of the datasets used. Datasets that differ significantly in domain or conversation style might pose challenges.
  • Ambiguity of Certain Emotions: Emotions like "trust" can be difficult to categorize definitively, and the mapping relies on empirical accuracy rather than absolute theoretical certainty.
  • Limited Comparative Studies: As this is a novel approach, there are few existing studies to directly compare against.

Future work could involve expanding the mapping to include more emotion categories, exploring its effectiveness on a wider range of datasets and modalities, and further investigating the psychological underpinnings of emotion categorization differences. The authors hope their work serves as a foundation for future research in this area.

Summary updated

BibTeX
@inproceedings{gong2024mapping,
  author = {Ziwei Gong and Muyin Yao and Xinyi Hu and Xiaoning Zhu and Julia Hirschberg},
  title = {{A Mapping on Current Classifying Categories of Emotions Used in Multimodal Models for Emotion Recognition}},
  booktitle = {EACL},
  year = {2024},
}
Figure