Transformer based ensemble for emotion detection

Aditya Kane, Shantanu Patankar, Sahil Khose, Neeraja KirtaneACL 2022

TL;DR

This paper tackles the challenge of accurately detecting nuanced emotions in essays for the WASSA 2022 shared task. The authors propose an ensemble of transformer models (BERT and ELECTRA) enhanced by novel data augmentation techniques to address class imbalance, achieving a competitive F1 score.

What Problem Does This Paper Address?

Accurately understanding and detecting human emotions expressed in text is a fundamental challenge in natural language processing, crucial for building more empathetic and interactive AI systems. While sentiment analysis has been extensively studied, identifying specific emotions like sadness, surprise, neutral, anger, fear, and disgust at an essay level is more complex. This paper addresses the WASSA 2022 shared task on empathy detection and emotion classification, specifically focusing on classifying emotions within essays. A key challenge in this domain is the inherent class imbalance in emotion datasets, where some emotions are far more prevalent than others, making it difficult for models to learn less frequent emotions effectively.

Methods

The core of the proposed solution lies in an ensemble of transformer-based language models, specifically BERT and ELECTRA. These models are chosen for their strong performance in various NLP tasks. To combat the class imbalance issue, the authors introduce and experiment with three data augmentation techniques:

  • Augmented Over-UnderSampling (AOUS): This method undersamples majority classes and oversamples minority classes by adding Reddit comments (sorted by length) from the GoEmotions dataset.
  • Random Synthetic Oversampling (RSO): This technique creates synthetic essays by concatenating random comments from the GoEmotions dataset, aiming to match the length distribution of the original WASSA dataset.
  • Augmented Oversampling (AOS): Similar to AOUS, this method oversamples minority classes by adding comments from GoEmotions with the longest lengths.

The final model is an ensemble of the best-performing individual models (BERT with AOUS, ELECTRA with AOS, ELECTRA with RSO, and ELECTRA with AOUS) using a majority voting mechanism.

Datasets

  • WASSA 2022 Shared Task Dataset: This is the primary dataset used for training and evaluation, consisting of essays labeled with seven emotions: anger, disgust, fear, joy, neutral, sadness, and surprise. The dataset has 1860 data points, with validation and test splits of 270 and 525 data points, respectively.
  • GoEmotions Dataset: This dataset, containing 27 emotions, was utilized for data augmentation to address the class imbalance in the WASSA dataset.

Key Insights and Ideas

  • Data Augmentation is Crucial: The paper strongly emphasizes that data augmentation techniques are essential for handling class imbalance in emotion detection tasks. Models trained on augmented datasets significantly outperform those trained on the original imbalanced data.
  • ELECTRA's Edge: ELECTRA, a variation of BERT, generally performed better than BERT across various augmented datasets, suggesting its efficiency and effectiveness for this task.
  • Ensembling for Robustness: Combining multiple well-performing models through majority voting leads to a more robust and accurate final prediction, as evidenced by the ensemble's superior performance.
  • Surprise and Joy Distinction: Contrary to expectations, the models performed well in distinguishing between the positive emotions "joy" and "surprise," suggesting that these emotions have distinct linguistic markers in the corpus.
  • RSO's Potential Drawbacks: While RSO aims to preserve length distribution, the random concatenation of comments might disrupt essay context, leading to less effective learning for some emotions.

Key Results

The authors achieved an F1 score of 62.76% on the test set using their ensemble model. Individual models showed varying performance, with ELECTRA fine-tuned on the AOUS dataset achieving a macro F1 of 59.67% on validation. The ensemble model demonstrated significant gains, particularly for the "neutral," "sadness," and "surprise" emotions. The paper also highlights that a batch size of 8 outperformed larger batch sizes (32 or 64), likely due to improved generalization from increased stochasticity.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Class Imbalance Mitigation: While data augmentation helped, the inherent class imbalance remains a challenge. Further investigation into more sophisticated sampling techniques could be beneficial.
  • Multi-task Learning: The paper briefly explored multi-task learning (combining empathy and emotion classification) but found it difficult to converge. Future work could focus on more robust multi-task learning architectures.
  • Pretraining on GoEmotions: Pretraining models on the larger GoEmotions dataset before fine-tuning on the WASSA dataset is suggested as a potential avenue for improvement.
  • Spurious Behavior: The paper notes that "anger" and "disgust" labels exhibit some spurious behavior, and further research is needed to understand and rectify this.
  • RSO Data Disruption: The random concatenation in RSO might disrupt essay context, impacting performance. Investigating more context-aware synthetic data generation is a promising direction.

Summary updated

BibTeX
@inproceedings{kane2022transformer,
  author = {Aditya Kane and Shantanu Patankar and Sahil Khose and Neeraja Kirtane},
  title = {{Transformer based ensemble for emotion detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure