Fraunhofer SIT at WASSA 2024 Empathy and Personality Shared Task: Use of Sentiment Transformers and Data Augmentation With Fuzzy Labels to Predict Emotional Reactions in Conversations and Essays

Raphael Frick, Martin SteinebachACL 2024

TL;DR

This paper presents Fraunhofer SIT's approach to the WASSA 2024 Empathy and Personality Shared Task, focusing on predicting emotional reactions to harmful news content. They achieved top rankings by combining LLM-driven data augmentation with fuzzy labels and fine-tuning sentiment transformers, demonstrating the effectiveness of specialized models and data enrichment for nuanced emotional understanding.

What Problem Does This Paper Address?

Predicting human emotional responses, particularly empathy, to online content is a significant challenge for AI. This is especially true when dealing with harmful content, which can evoke complex and varied reactions. The WASSA 2024 Empathy and Personality Shared Task aimed to push the boundaries of AI's ability to understand these nuanced emotional responses. Participants were tasked with analyzing news articles containing harmful content and predicting how individuals would react, either through written essays or conversational dialogues. This involves not only identifying emotions but also understanding the underlying empathy and personality traits that influence these reactions. The task highlights the need for AI systems that can go beyond simple sentiment analysis to grasp the deeper emotional landscape of human interaction with online information.

Methods

The Fraunhofer SIT team employed a multi-pronged approach combining data augmentation with fine-tuned sentiment transformer models.

  • Data Augmentation with Fuzzy Labels: To overcome the limitations of small, potentially imbalanced datasets, they utilized LLM-driven data augmentation. This involved paraphrasing and back-translation of existing text samples. Crucially, they introduced "fuzzy labels" for the augmented data. This means that instead of assigning exact labels, they added a small degree of noise (e.g., +/- 0.2 for Task 3, +/- 0.1 for Task 2) to the original labels. The rationale was that slight mistranslations or rephrasing might not perfectly preserve the original semantic meaning, and thus, the labels should reflect this uncertainty. This approach aimed to create more robust models that are less prone to overfitting on specific phrasing.
  • Fine-tuned Sentiment Transformers: The core of their predictive models were transformer architectures, specifically RoBERTa-based models. They experimented with several pre-trained models, including DeBERTa V3 Large, SieBERT (fine-tuned on multiple sentiment datasets), and Twitter RoBERTa Base Sentiment. The key insight here was that models pre-trained or fine-tuned on sentiment-specific data (like SieBERT and Twitter RoBERTa) often outperformed more general-purpose large language models like DeBERTa. This suggests that domain-specific pre-training is highly beneficial for tasks requiring nuanced emotional understanding.

Datasets

The primary dataset used was introduced for the second iteration of the WASSA Empathy and Personality Shared Task. This dataset includes:

  • News articles containing harmful content.
  • Written essays from individuals reacting to these articles, annotated with Batson empathic concern and personal distress scores, as well as Big Five personality traits (OCEAN).
  • Conversations between two users who read the same article, with each speech turn annotated for perceived empathy, emotion polarity, and intensity.
  • Person-level demographic information.

The paper also mentions using existing sentiment estimation datasets for fine-tuning the SieBERT model.

Key Insights and Ideas

  • Fuzzy Labels Enhance Robustness: Introducing noise to labels of augmented data, rather than using exact labels, proved beneficial. This "fuzzy" approach helps models generalize better and avoid overfitting to specific linguistic variations introduced by augmentation.
  • Sentiment-Specific Pre-training is Key: Models fine-tuned on sentiment-rich datasets (like SieBERT on reviews and tweets) significantly outperformed general-purpose large language models (like DeBERTa) on empathy and emotion prediction tasks. This highlights the importance of domain-specific knowledge for these nuanced tasks.
  • Twitter Data May Be Too Stylistically Different: The Twitter RoBERTa Base Sentiment model, despite its specialized training, did not perform as well as SieBERT. This suggests that the unique stylistic characteristics of tweets might not transfer optimally to predicting emotions in essays or dialogues, which have different linguistic conventions.
  • Empathy Prediction Remains Challenging: Even with top-ranking performance, the Pearson Correlation Coefficients for empathy prediction, particularly at the dialogue level (Task 1), were noted as being low. This indicates that accurately capturing the subtle nuances of empathy in conversations is an ongoing research frontier.

Key Results

The Fraunhofer SIT team achieved impressive results in the WASSA 2024 Shared Task:

  • Task 1 (CONVD - Empathy Prediction in Conversations): 1st place. The SieBERT-based model achieved a Pearson Correlation Coefficient of 0.193 on the test set.
  • Task 2 (CONVT - Empathy and Emotion Prediction in Conversation Turns): 8th place. While the DeBERTa model showed promise on validation, the final performance on the test set was lower, suggesting challenges with short context windows.
  • Task 3 (EMP - Empathy Prediction in Essays): 3rd place. The SieBERT model achieved a Pearson Correlation Coefficient of 0.6871 for empathy and 0.684 for emotional distress on the development set, and ranked third overall.

The paper also shows that data augmentation, even with fuzzy labels, consistently improved performance across tasks.

Limitations and Future Work

  • Low Correlation for Dialogue Empathy: The authors acknowledge that the low Pearson Correlation Coefficient for Task 1, despite achieving first place, indicates that current models struggle to predict meaningful labels accurately at the dialogue level. Label imbalance is cited as a significant factor.
  • Seed Dependency: Model performance can vary significantly depending on the random seeds used during training.
  • Stylistic Mismatch: The performance of Twitter-trained models suggests that stylistic differences between tweet data and essay/dialogue data can hinder transferability. Future work should focus on sentiment transformers trained on more diverse or relevant data sources.
  • Context Window Limitations: For conversational turn prediction (Task 2), short context windows may not capture sufficient information. Future research could explore combining larger context windows and integrating information from both turn and dialogue levels.
  • Regression vs. Classification: The paper notes that transitioning from a regression to a classification problem for empathy prediction did not fully resolve the issues related to label imbalance. Further exploration into handling imbalanced datasets is warranted.

Summary updated

BibTeX
@inproceedings{frick2024fraunhofer,
  author = {Raphael Frick and Martin Steinebach},
  title = {{Fraunhofer SIT at WASSA 2024 Empathy and Personality Shared Task: Use of Sentiment Transformers and Data Augmentation With Fuzzy Labels to Predict Emotional Reactions in Conversations and Essays}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure