Improved Text Emotion Prediction Using Combined Valence and Arousal Ordinal Classification

Michail Mitsios, Georgios Vamvoukakis, Georgia Maniati, Nikolaos Ellinas, Georgios Dimitriou, Konstantinos Markopoulos, Panos Kakoulidis, Alexandra Vioni, Myrsini Christidou, Junkwang Oh, Gunu Jho, Inchul Hwang, Georgios Vardaxoglou, Aimilios Chalamandaris, Pirros Tsiakoulis, Spyros RaptisNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of accurately predicting emotions from text by moving beyond traditional discrete classification. They introduce an ordinal classification approach that considers the inherent relationships between emotions based on valence and arousal, leading to more meaningful predictions and reduced error severity, even while maintaining high accuracy.

What Problem Does This Paper Address?

Accurately understanding human emotions expressed in text is crucial for building more empathetic and natural human-computer interaction systems, from chatbots to text-to-speech. However, current methods often treat emotions as distinct, unrelated categories. This can lead to problematic misclassifications where, for instance, mistaking "sadness" for "joy" is considered as severe an error as mistaking it for "depression." This lack of nuance fails to capture the perceptual similarities between emotions, resulting in unnatural or misleading outputs in applications. The paper aims to address this by developing a model that acknowledges and leverages the ordered nature of emotional states.

Methods

The authors propose a multi-stage approach. First, they establish a strong baseline using a RoBERTa-CNN model, a standard transformer-based architecture known for its effectiveness in text analysis, for traditional discrete emotion classification.

The core innovation lies in their shift to ordinal classification. Instead of treating each emotion as a separate class, they arrange emotions along a valence scale (from negative to positive) and then extend this to a two-dimensional space incorporating arousal (from low to high). This ordinal approach is implemented by:

  1. Ordinal Valence Classification: Emotions are ordered based on their valence, and the model is trained to predict this order rather than a discrete label. This is achieved by replacing the standard cross-entropy loss with a Mean Square Error (MSE) loss, which encourages the model to minimize the distance between predicted and target valence values.
  2. 2D Ordinal Classification (Valence and Arousal): To further refine emotion prediction, they extend the ordinal approach to a 2D space, mapping emotions onto a grid defined by valence and arousal. This involves training two separate classifier heads simultaneously – one for valence and one for arousal – and combining their losses. The predicted valence and arousal values then serve as coordinates within this emotion grid.

Datasets

The study utilizes three publicly available datasets commonly used in emotion recognition research:

  • ISEAR: A balanced dataset with 7 distinct emotion labels (joy, anger, sadness, shame, guilt, surprise, fear).
  • Wassa-21: Contains essays expressing empathy and distress in response to news articles.
  • GoEmotions: A large dataset of Reddit comments annotated with 27 emotions or a neutral label.

For comparability, the authors pre-processed these datasets to align with Ekman's basic emotions where applicable.

Key Insights and Ideas

  • Emotions are not discrete: The paper strongly argues that treating emotions as entirely separate categories is suboptimal. The perceptual similarities between emotions (e.g., excitement and amusement sharing positive valence) are crucial for accurate prediction.
  • Ordinal classification reduces error severity: Even if overall accuracy doesn't dramatically increase, ordinal classification leads to "less severe mistakes." This means misclassifications are more likely to occur between closely related emotions rather than wildly different ones, which is critical for applications like TTS.
  • Valence and Arousal provide a richer emotional space: Moving beyond a single dimension (valence) to a 2D space (valence and arousal) significantly improves the model's ability to differentiate a wider range of emotions, especially when dealing with datasets with many fine-grained labels.
  • Ordinal approach aids in unseen data: The 2D ordinal model demonstrated an ability to correctly classify emotions it hadn't explicitly seen during training, by placing them in proximity to their ground truth location in the valence-arousal space.

Key Results

  • The baseline RoBERTa-CNN model achieved state-of-the-art performance on standard emotion classification tasks.
  • The ordinal classification approach, while maintaining comparable accuracy and F1-scores to the baseline on ISEAR and Wassa-21, significantly reduced the magnitude of misclassification errors. For instance, on ISEAR, the ordinal model preferred misclassifications with a valence distance of 1 over errors with distances larger than 3.
  • For the more complex GoEmotions dataset with 23 emotions, the 2D ordinal classification model showed a substantial improvement in F1-score (0.63) and accuracy (0.52) compared to the baseline (0.12 F1-score, 0.28 accuracy).

Limitations and Future Work

Limitations:

  • The paper notes that on the Wassa-21 dataset, the ordinal model did not achieve a macro-F1 score comparable to the baseline, attributing this to the dataset's imbalance and the limitations of MSE in handling such cases.
  • The 2D ordinal classification was evaluated on a subset of 23 emotions from GoEmotions, ensuring each grid cell corresponded to at most one emotion.

Future Work:

  • Extending the research to diverse datasets.
  • Exploring alternative models and different emotion ordering schemes.
  • Investigating methods to interpret the model's components to better understand feature importance and further improve the method.

Summary updated

BibTeX
@inproceedings{mitsios2024improved,
  author = {Michail Mitsios and Georgios Vamvoukakis and Georgia Maniati and Nikolaos Ellinas and Georgios Dimitriou and Konstantinos Markopoulos and Panos Kakoulidis and Alexandra Vioni and Myrsini Christidou and Junkwang Oh and Gunu Jho and Inchul Hwang and Georgios Vardaxoglou and Aimilios Chalamandaris and Pirros Tsiakoulis and Spyros Raptis},
  title = {{Improved Text Emotion Prediction Using Combined Valence and Arousal Ordinal Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.72},
}
Figure