TL;DR
This paper presents a transformer-based approach for predicting empathy and classifying emotions in text, specifically in response to news articles. By fine-tuning the RoBERTa model and employing data augmentation techniques, the authors achieved competitive results on the WASSA 2022 shared task, demonstrating the effectiveness of large pre-trained models for these nuanced NLP tasks.
What Problem Does This Paper Address?
Understanding and predicting human empathy and emotional responses to textual content is a complex but crucial area in natural language processing. This paper tackles the challenge of modeling empathy and emotion classification in reactions to news articles, particularly those depicting disturbing or harmful situations. The goal is to quantify an individual's empathic concern, personal distress, and the specific emotions they express when engaging with such content. This has implications for mental health analysis, content moderation, and building more emotionally intelligent AI systems.
Methods
The core of the proposed system relies on fine-tuning the RoBERTa transformer model. For empathy prediction (Task 1), two separate RoBERTa models were trained: one to predict empathy scores and another for distress scores. The input essays were tokenized using the RoBERTa tokenizer, and the models were trained with a Mean Squared Error (MSE) loss function. Notably, all model parameters were trainable, and five additional layers (four linear and one dropout) were added on top of RoBERTa to deepen the network. Five demographic features were also incorporated as input to the pre-final layer.
For emotion classification (Task 2), a similar RoBERTa fine-tuning approach was used, but with the output layer adjusted to predict seven emotion labels (the six basic emotions from Ekman plus neutral). The loss function here was Cross-Entropy Loss.
A key methodological contribution is the use of data augmentation via back-translation. Essays were translated to another language and then back to English to generate paraphrased versions, effectively doubling the training data. This was particularly important for addressing data imbalance in the emotion classification task.
Datasets
The study utilizes a dataset that is an extension of the one provided by Buechel et al. (2018). This dataset comprises essays written by participants who had read news articles about people or disturbing situations. For each essay, the dataset includes:
- Empathy score
- Distress score
- Emotion label (six basic emotions + neutral)
- Demographic information (age, gender, race, education, income)
- Personality information (conscientiousness, openness, extraversion, agreeableness, stability)
- Interpersonal Reactivity Index (IRI) scores (fantasy, perspective taking, empathetic concern, personal distress)
The dataset was split into train, development, and test sets with 1860, 270, and 525 examples, respectively.
Key Insights and Ideas
- RoBERTa's Efficacy: The paper reinforces the power of large pre-trained transformer models like RoBERTa for complex NLP tasks such as empathy and emotion prediction, even when fine-tuned on relatively smaller, specialized datasets.
- Data Augmentation's Impact: Back-translation proved to be a valuable technique for augmenting the dataset, leading to improved performance, particularly in the emotion classification task. This highlights the importance of data quantity and diversity for robust model training.
- Handling Imbalance: The authors explicitly addressed data imbalance in emotion classification by using random over-sampling, a crucial step for preventing models from being biased towards majority classes.
- Demographic Features: Incorporating demographic features alongside textual input for empathy prediction suggests that these auxiliary features can provide valuable context for understanding emotional responses.
Key Results
- Task 1 (Empathy Prediction): The system achieved a Pearson Correlation Coefficient of 0.488 on the test set. This score is an average of the empathy score (0.470) and distress score (0.506).
- Task 2 (Emotion Classification): The system achieved a Macro F1-score of 0.531 on the test set.
- Data Augmentation: Table 2 shows that data augmentation generally improved metrics for emotion classification, with Macro F1-score increasing from 0.5174 to 0.5311 and Macro Precision from 0.5461 to 0.557.
Limitations and Future Work
- Overfitting: The authors noted that the model experienced overfitting, particularly on the empathy score during development, leading to a slightly lower score on the test set compared to the development set for distress. Early stopping was used to mitigate this for distress.
- Dataset Size: While data augmentation helped, the inherent size of the dataset might still be a limiting factor for achieving even higher performance on such nuanced tasks.
- Future Work: The paper implicitly suggests that further exploration of data augmentation techniques and potentially more sophisticated model architectures could lead to further improvements. The inclusion of demographic and personality features also opens avenues for more personalized empathy prediction models.
Summary updated