WASSA 2022 Shared Task: Predicting Empathy, Emotion and Personality in Reaction to News Stories

Valentin Barriere, Shabnam Tafreshi, João Sedoc, Sawsan AlqahtaniACL 2022

TL;DR

This paper presents the WASSA 2022 Shared Task, which aimed to predict empathy, emotion, and personality from essays written in reaction to news stories. The task highlights the growing interest in understanding nuanced human responses to text and provides a benchmark for future research in this area.

What Problem Does This Paper Address?

Understanding and predicting human emotional and psychological responses to textual content is a complex but crucial task. While emotion detection has seen significant progress, predicting more nuanced constructs like empathy and personality from text remains challenging. This paper addresses the need for standardized evaluation and datasets for these tasks, particularly in the context of reactions to news stories that describe harm. Such predictions have applications in areas like mental health support, human-robot interaction, and content moderation.

Methods

The WASSA 2022 Shared Task involved four tracks: predicting empathy and distress scores, predicting emotion categories, predicting personality traits (Big Five OCEAN model), and predicting interpersonal reactivity (IRI). Participants were encouraged to develop models using various machine learning architectures, with a strong emphasis on transformer-based models like RoBERTa and DeBERTa. Features used included the essay text itself, demographic information (age, gender, ethnicity, income, education), and personality traits. Teams were allowed to use external resources and pre-trained models.

Datasets

The core dataset used for the shared task was an extension of previous work, comprising news articles describing harm to individuals or groups, paired with essays written by participants reacting to these articles. This dataset was further enriched with emotion labels (Ekman's six basic emotions plus "no-emotion") and supplemented with author demographics and personality information. The dataset was split into training, development, and testing sets.

Key Insights and Ideas

  • The Power of Contextualized Embeddings: Transformer models, particularly those fine-tuned on large datasets, proved highly effective across all tracks, demonstrating their ability to capture complex linguistic nuances.
  • Demographics and Personality Matter: Incorporating demographic and personality information as features significantly boosted performance, suggesting that individual differences play a crucial role in how people react to news.
  • Joint Modeling is Beneficial: Several top-performing systems achieved better results by jointly modeling empathy and distress, indicating a strong correlation between these two constructs.
  • Emotion is Multifaceted: The emotion prediction task highlighted the difficulty of assigning a single label to an essay, as multiple emotions can be present, and even sentence-level emotions can vary.
  • Data Augmentation is Key: Using external datasets for pre-training and data augmentation techniques (like random punctuation) proved beneficial, especially for emotion modeling, even when the external data genre differed from the task's.

Key Results

  • Empathy Prediction: The best system achieved an average Pearson correlation of 0.540. This was a notable improvement over previous benchmarks (r=.404 for empathy and r=.444 for distress in Buechel et al., 2018).
  • Emotion Prediction: The top system achieved a macro F1-score of 69.8% and an accuracy of 75.4%. Sadness and anger were the most frequently predicted and classified emotions.
  • Personality and IRI Prediction: The best results for personality prediction were an average Pearson correlation of 0.230, and for IRI, it was 0.255. One team notably achieved strong results by aggregating predictions across multiple essays from the same user.
  • Error Analysis: The analysis revealed that models sometimes struggled with essays exhibiting a mix of empathy and distress, or scenarios where empathy was present without distress, and vice versa.

Limitations and Future Work

The paper acknowledges that the distribution of emotion tags in the dataset is imbalanced, with sadness being the most prevalent. The authors also note that assigning a single emotion label at the document level can be challenging due to the presence of multiple emotions within an essay. Future work could focus on developing more robust models for handling imbalanced data and exploring multi-label emotion classification at the sentence or essay level. The availability of sentence-level emotion labels for the entire dataset is also a promising avenue for future research. The success of incorporating demographic and personality features suggests further exploration into how these factors interact with textual content to influence emotional responses.

Summary updated

BibTeX
@inproceedings{barriere2022wassa,
  author = {Valentin Barriere and Shabnam Tafreshi and João Sedoc and Sawsan Alqahtani},
  title = {{WASSA 2022 Shared Task: Predicting Empathy, Emotion and Personality in Reaction to News Stories}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure