Findings of WASSA 2023 Shared Task on Empathy, Emotion and Personality Detection in Conversation and Reactions to News Articles

Valentin Barriere, João Sedoc, Shabnam Tafreshi, Salvatore GiorgiACL 2023

TL;DR

This paper presents the WASSA 2023 shared task, which focused on automatically detecting empathy, emotion, and personality in text. The task utilized a novel dataset of reactions to news articles, including essays and subsequent conversations, and involved five distinct prediction tracks. The findings highlight the continued dominance of transformer-based models and the challenges in achieving high performance across all tasks, particularly in capturing nuanced emotional states and personality traits.

What Problem Does This Paper Address?

Understanding and detecting human emotions, empathy, and personality in text is crucial for building more effective and human-like AI systems. This is particularly important in conversational AI, where systems need to respond appropriately to user sentiment and emotional states. While previous work has explored these areas, there's a need for standardized benchmarks and datasets that capture these phenomena in more complex, multi-turn interactions and in the context of reactions to real-world events like news articles. This paper addresses this by presenting the WASSA 2023 shared task, which aims to advance research in these areas by providing a challenging dataset and evaluating a variety of computational approaches.

Methods

The WASSA 2023 shared task involved five tracks, each focusing on a different prediction task:

  • Turn-level Empathy and Emotion in Conversations (CONV): Predicting emotional polarity, intensity, and perceived empathy for each turn in a conversation. This is a new track for WASSA 2023, leveraging conversational data.
  • State Empathy Prediction (EMP): Predicting self-reported empathic concern and personal distress scores from essays written in response to news articles.
  • Emotion Label Prediction (EMO): Predicting categorical emotion tags (including Ekman's six basic emotions, plus hope and neutral) for essays.
  • Personality Prediction (PER): Predicting Big Five (OCEAN) personality traits from essays.
  • Multi-dimensional Trait Empathy Prediction (IRI): Predicting scores for four subscales of the Interpersonal Reactivity Index (perspective taking, fantasy, empathic concern, and personal distress) from essays.

Participants were encouraged to use transformer-based models (e.g., BERT, RoBERTa, DeBERTa) and were allowed to leverage external resources and pre-trained models. Some teams also explored multi-task learning and interaction-aware models.

Datasets

The primary dataset for the WASSA 2023 shared task was derived from Omitaomu et al. (2022). This dataset was extended with additional essay-level emotion annotations. The dataset includes:

  • News articles that express harm to an entity.
  • Crowdsourced essays written in reaction to these articles.
  • Conversations between crowd workers about the news articles.
  • Self-report data from crowd workers on personality (Big Five), empathy (IRI), empathic concern, and personal distress.
  • Third-party annotations for conversational turns (empathy, emotion polarity, emotion intensity) and essays (multi-label emotions).

This dataset builds upon previous WASSA shared task datasets but introduces the novel element of conversational data.

Key Insights and Ideas

  • Transformer Dominance: As in previous years, transformer-based models, particularly those based on BERT, RoBERTa, and DeBERTa, were the backbone of the top-performing systems across most tracks. This underscores their effectiveness in capturing complex linguistic patterns.
  • Context is Key, But Underutilized: While the dataset provided conversational context, surprisingly few teams explicitly leveraged speaker identifiers or multi-task learning across tracks to integrate this contextual information effectively. This suggests a significant opportunity for future research.
  • Empathy and Emotion are Intertwined but Distinct: The task highlights the close relationship between empathy and emotion, but also the challenges in disentangling them. The inclusion of both state and trait empathy measures, along with various emotion categories, allowed for a nuanced exploration.
  • Personality Prediction Remains Challenging: The PER track saw some participants reporting negative correlations for certain personality traits, indicating the difficulty in accurately predicting personality from text alone, especially with limited data.
  • External Data Boosts Performance: Leveraging external datasets for pre-training or as features proved beneficial for many teams, demonstrating the value of broader knowledge bases.

Key Results

  • CONV Track: The best system, HIT-SCIR, achieved an average Pearson correlation of .758 across emotion polarity, intensity, and perceived empathy.
  • EMP Track: NCUEE-NLP led with an average Pearson correlation of .418 for empathy and distress prediction. This performance is comparable to previous WASSA tasks but slightly lower than the best systems from prior years when considering larger training sets.
  • EMO Track: Adityapatkar achieved the highest Macro F1 score of 70.1% and micro-Jaccard of 60.1% for emotion prediction. Fear, neutral, and sadness were among the most predictable emotions.
  • PER/IRI Tracks: YNU-HPCC achieved the best results for both personality (PER) and multi-dimensional trait empathy (IRI) prediction. However, the PER track also saw notable negative correlations, suggesting significant challenges.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the test dataset size for tracks 2-5 might limit the generalizability of conclusions.
  • Domain Shift: Models trained on general text might not perform optimally on reaction essays, which can be influenced by specific news content and personal biases.
  • Subjectivity of Annotation: Annotating empathy and emotions from text alone (without speech cues or body language) is inherently subjective and can be influenced by annotator biases.
  • Future Work: The paper suggests that more sophisticated interaction-aware models and multi-task learning approaches could improve performance. There's also a call for more psycho-linguistic work to better define and differentiate empathy and emotion. The potential for misuse of these models for manipulation is also highlighted as an ethical concern.

Summary updated

BibTeX
@inproceedings{barriere2023findings,
  author = {Valentin Barriere and João Sedoc and Shabnam Tafreshi and Salvatore Giorgi},
  title = {{Findings of WASSA 2023 Shared Task on Empathy, Emotion and Personality Detection in Conversation and Reactions to News Articles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure