VISU at WASSA 2023 Shared Task: Detecting Emotions in Reaction to News Stories Leveraging BERT and Stacked Embeddings

Vivek Kumar, Sushmita Singh, Prayag TiwariarXiv 2023 · DOI · arXiv

Introduction

Detecting emotions in text is a nuanced challenge that extends far beyond binary sentiment analysis. At the WASSA 2023 Shared Task, the VISU team addressed Track 3: Emotion Classification (EMO), which requires predicting emotions from long-form essays (300–800 words) written in response to news articles. This task presents two primary hurdles: extreme granularity, with 31 distinct emotion categories (including complex mixtures like Hope/Sadness), and severe data scarcity.

Dataset and Challenges

The study utilized the WASSA 2023 Shared Task dataset, comprising only 1,000 samples split into 792 for training, 208 for development, and 100 for testing. The distribution of labels is highly skewed; while "Sadness" and "Neutral" are relatively well-represented, many "mixed" categories have fewer than five examples, or even just one, in the training set.

Train vs. Dev Dataset Distribution showing the extreme imbalance across 31 emotion categories.

Methodology

The researchers evaluated two distinct architectural approaches to handle this complex classification task:

  1. BiLSTM with Stacked Embeddings: This architecture utilizes a Bidirectional Long Short-Term Memory (BiLSTM) network. To capture diverse linguistic features, the team employed "stacked embeddings," combining GloVe (static semantic meaning), fastText (sub-word information), and BERT (contextual representations) into a single input vector.
  2. Fine-tuned BERT-Base: A standard Transformer-based model was fine-tuned with additional dense layers. This approach leverages the self-attention mechanism to capture long-range dependencies within the essays.

The preprocessing pipeline included expanding contractions and removing special characters. Interestingly, the authors noted that stemming and lemmatization—common NLP practices—actually degraded performance, suggesting that morphological nuances are vital for emotional expression.

Experimental Results

The experimental results highlighted a significant performance gap between recurrent architectures and Transformer-based models. While stacking embeddings improved the BiLSTM's performance over static embeddings, it could not compete with the fine-tuned BERT model.

Performance comparison of different deep learning models across various metrics.

As shown in the results table, the BERT-base model achieved a Micro F1-Score of 0.593, nearly tripling the 0.213 achieved by the best BiLSTM variant. In the official competition, the VISU system secured 10th place with a Macro F1-Score of 0.2717. The low Macro F1-Score across all models underscores the difficulty of generalizing to the underrepresented minority emotion classes.

Conclusion

The VISU team's participation in WASSA 2023 confirms that contextual embeddings are superior for long-form emotional analysis, yet data imbalance remains a critical bottleneck. Future work will focus on data augmentation techniques to bolster minority classes and exploring domain-specific adaptations to better interpret the context of news-reaction dialogues.

Summary updated

BibTeX
@misc{kumar2023visu,
  author = {Vivek Kumar and Sushmita Singh and Prayag Tiwari},
  title = {{VISU at WASSA 2023 Shared Task: Detecting Emotions in Reaction to News Stories Leveraging BERT and Stacked Embeddings}},
  year = {2023},
  eprint = {2307.15164},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.15164},
}
Figure