TL;DR
This paper details the SINAI team's participation in the WASSA 2022 shared task, focusing on predicting empathy, distress, and emotions from text. They achieved impressive results by combining state-of-the-art Transformer models with novel ensemble strategies, ranking 2nd and 3rd in their respective tracks.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically understanding complex human emotional states from text, specifically empathy, distress, and discrete emotions. While emotion analysis is a well-established NLP task, empathy (the ability to sense and imagine others' feelings) and distress (a self-focused negative affective state) are less explored. Accurately predicting these nuanced states has significant implications for mental health support, understanding social interactions, and analyzing reactions to news events. The WASSA 2022 shared task provided a benchmark for these tasks, requiring models to predict empathy and distress scores and classify emotions in user reactions to news articles.
Methods
The SINAI team employed a multi-pronged approach for the two tracks:
- Track 1: Empathy and Distress Prediction: They explored two main strategies:
- Multi-output Regression: A single Transformer model (RoBERTa-large) was fine-tuned to predict both empathy and distress scores simultaneously.
-
Separated Regression Models: Two distinct models were trained, one for empathy and one for distress. They experimented with different Transformer architectures (RoBERTa, BERT, DistilBERT) and investigated the impact of pre-training on emotion datasets versus using raw language models. They also explored incorporating demographic features.
-
Track 2: Emotion Classification: This track involved a multi-class classification task. The team investigated:
- Supervised Learning: Fine-tuning a RoBERTa-base model on the provided data.
- Zero-Shot Learning (ZSL): Leveraging Natural Language Inference (NLI) models (DeBERTa-xlarge-mnli) and a large language model (GPT-3, specifically OpenAI's Davinci) to classify emotions without direct training on the task.
- Ensemble Methods: Combining the predictions of supervised and ZSL models, with a tie-breaking mechanism favoring the supervised model.
Datasets
The primary dataset used was an extension of the one presented in Buechel et al. (2018) for the WASSA 2022 shared task. This dataset consists of posts reacting to news articles where harm to a person, group, or other is present. It includes:
- Regression Scores: For empathy and distress (1-7 scale) for Track 1.
- Emotion Categories: Seven categories (anger, fear, sadness, joy, disgust, surprise, and neutral) for Track 2.
- Demographic Information: Person-level data such as age, gender, ethnicity, income, and education level.
The paper also mentions fine-tuning on the GoEmotions dataset (for BERT) and the CARER dataset (for DistilBERT), which contain emotion labels.
Key Insights and Ideas
- Raw Language Models Outperform Emotion-Tuned Models for Empathy/Distress: Surprisingly, for empathy and distress prediction (Track 1), a raw RoBERTa-large model, especially when augmented with demographic features, performed better than models pre-trained on emotion-specific datasets. This suggests that direct emotion knowledge might not always be beneficial for predicting these related but distinct concepts.
- ZSL Models Show Promise for Emotion Classification: Zero-shot learning models (NLI and GPT-3) achieved competitive results in emotion classification (Track 2) without being explicitly trained on the task. This highlights the potential of ZSL for scenarios with limited labeled data.
- Ensembling is Key to Top Performance: For both tracks, ensemble methods proved crucial. Combining predictions from multiple models, including both supervised and ZSL approaches for emotion classification, led to the best overall performance.
- Demographic Features Can Be Helpful: Incorporating demographic information (gender, age, income) as features in the regression models for empathy and distress prediction improved performance, indicating that these personal attributes can provide valuable context.
Key Results
- Track 1 (Empathy & Distress): The SINAI team ranked 2nd overall. Their best model, a multi-output regression model based on RoBERTa-large with demographic features, achieved an average Pearson correlation of 0.53 on the test set.
- Track 2 (Emotion Classification): The team ranked 3rd overall. Their ensemble model achieved an accuracy of 0.636 and a macro F1-score of 0.553 on the test set.
Limitations and Future Work
- Demographic Feature Decoding: The authors faced challenges in accurately decoding categorical demographic features (race, education) and thus only utilized numerical ones (age, income) and inferred gender.
- ZSL Exploration: While ZSL models showed promise, the authors plan to further explore their potential for emotion classification as a future direction.
- Ensemble Strategy Refinement: The ensemble for emotion classification used a tie-breaking rule favoring the supervised model. Further research could explore more sophisticated ensemble weighting or combination strategies.
Summary updated