Converge at WASSA 2023 Empathy, Emotion and Personality Shared Task: A Transformer-based Approach for Multi-Label Emotion Classification

Aditya Paranjape, Gaurav Kolhatkar, Yash Patwardhan, Omkar Gokhale, Shweta DharmadhikariACL 2023

TL;DR

This paper presents a transformer-based approach for multi-label emotion classification on essays responding to news articles. By fine-tuning models like XLNet, Longformer, BERT, and BigBird, and employing ensembling techniques, the authors achieved a strong performance on the WASSA 2023 Empathy and Emotion Shared Task, highlighting the effectiveness of threshold-based voting for complex multi-label scenarios.

What Problem Does This Paper Address?

Understanding and contextualizing human emotions in text is a significant challenge in Natural Language Processing (NLP), crucial for improving human-computer interaction. While emotion classification has been explored, accurately identifying multiple emotions within a single text (multi-label classification) remains complex. This paper tackles this challenge within the context of essays written in response to news articles, a domain where nuanced emotional expression is common. The WASSA 2023 Empathy and Emotion Shared Task specifically aims to address this by providing a dataset and a benchmark for multi-label emotion classification.

Methods

The core of the approach involves fine-tuning several state-of-the-art transformer-based models for the multi-label classification task. The models explored include:

  • XLNet: An autoregressive pre-training technique that addresses some of BERT's limitations by using a Permutation Language Modelling objective for better bidirectional context understanding.
  • Longformer: Designed for processing long sequences of text, it employs a modified attention mechanism that scales linearly with input size, making it suitable for longer essays.
  • BERT: A foundational language representation model known for its strong performance on various NLP tasks.
  • BigBird: A BERT-like model that handles longer input sequences by using a combination of sparse, global, and random attention, requiring less computational power than standard self-attention.
  • ELECTRA: A pre-training method that uses a generator-discriminator setup, aiming for greater computational efficiency.
  • RoBERTa: An optimized version of BERT with improved hyperparameters and training strategies.

To address class imbalance in the dataset, oversampling was used, where samples from underrepresented classes were duplicated. To further boost performance, ensembling techniques were applied, combining the predictions of multiple models. Specifically, the paper explores:

  • Voting: A simple majority vote among the top models.
  • Averaging: Averaging the probability outputs of the top models.
  • Threshold-based voting: A more sophisticated approach that incorporates confidence thresholds to retain predicted labels, particularly effective for samples with multiple ground truth labels.

A threshold of 0.37 was used to determine label assignment for individual model predictions.

Datasets

The primary dataset used is from the WASSA 2023 Empathy and Emotion Shared Task. This dataset consists of essays written in response to news articles, with lengths ranging from 300 to 800 characters. It features 8 base emotions or labels: Anger, Hope, Sadness, Neutral, Disgust, Surprise, Joy, and Fear. The dataset is characterized by a significant class imbalance, with 'Sadness' being the most frequent and 'Joy' the least frequent class. The paper uses the provided training, development, and test sets for evaluation.

Key Insights and Ideas

  • Transformer models excel at multi-label emotion classification: The study demonstrates that fine-tuning powerful transformer architectures can yield competitive results on this challenging task.
  • Long-sequence models are crucial for essay-based emotion analysis: Longformer's superior performance as a standalone model highlights the importance of handling longer text inputs effectively when analyzing essays.
  • Ensembling significantly boosts performance: Combining predictions from multiple models, especially through threshold-based voting, leads to substantial improvements over individual model performance.
  • Threshold-based voting is key for multi-label complexity: The paper's most significant finding is that threshold-based voting, particularly with a carefully chosen set of top models, is more effective than simple voting or averaging for multi-label scenarios where multiple emotions can co-occur.
  • Oversampling can be detrimental: Counter-intuitively, the oversampling technique employed in this study led to a decrease in performance, suggesting that naive oversampling might not be suitable for this specific dataset or task.

Key Results

  • Official Phase: XLNet achieved a macro F1-score of 0.5649, securing rank 6 on the official leaderboard.
  • Post-Competition Phase (Standalone Models): Longformer emerged as the best standalone model with a macro F1-score of 0.6360.
  • Post-Competition Phase (Ensembling):
    • Threshold-based voting with the top 3 models (Longformer, BERT, BigBird) achieved the highest overall macro F1-score of 0.6605.
  • Average-based ensemble with the top 5 models also performed well, yielding a macro F1-score of 0.6098.
  • Oversampling: The oversampling experiment resulted in a significant drop in macro F1-score to 0.4653.

Limitations and Future Work

  • Oversampling Ineffectiveness: The paper acknowledges that oversampling yielded no performance improvement and even decreased performance. Further investigation is needed to understand why this occurred and to explore alternative or refined oversampling techniques.
  • Future Work: The authors plan to improve their oversampling score and combine it with the successful threshold-based voting strategy. This suggests a continued focus on refining data augmentation and ensemble methods for better multi-label emotion classification.

Summary updated

BibTeX
@inproceedings{paranjape2023converge,
  author = {Aditya Paranjape and Gaurav Kolhatkar and Yash Patwardhan and Omkar Gokhale and Shweta Dharmadhikari},
  title = {{Converge at WASSA 2023 Empathy, Emotion and Personality Shared Task: A Transformer-based Approach for Multi-Label Emotion Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure