TL;DR
This paper presents a novel ensemble approach using sentiment-enhanced RoBERTa transformers for predicting empathy and emotion in text. Their system achieved top performance in the WASSA 2023 shared task, particularly excelling at the essay-level empathy and distress prediction.
What Problem Does This Paper Address?
The paper tackles the challenge of computationally detecting and predicting empathy and emotion in human communication. Empathy, the ability to understand or feel another's perspective, is a complex cognitive and emotional process. While previous work has explored various methods for its detection, this paper focuses on improving accuracy in predicting perceived empathy and emotion intensity, both at the granular speech-turn level within conversations and at the broader essay level. The WASSA 2023 shared task, specifically Track 1 (Empathy and Emotion Prediction in Conversations) and Track 2 (Empathy Prediction at the essay-level), provides a benchmark for evaluating these advancements.
Methods
The core of the NCUEE-NLP system is a unified architecture that ensembles three pre-trained RoBERTa-based transformer models:
- Original RoBERTa: A robustly optimized BERT pre-training approach.
- RoBERTa-Twitter: A RoBERTa model specifically fine-tuned on a large corpus of tweets for sentiment analysis.
- EmoBERTa: A RoBERTa model designed for emotion recognition in conversations, capable of learning speaker-aware states.
These models are individually fine-tuned on the task-specific datasets. For Track 1, each transformer is fine-tuned separately for empathy, emotion polarity, and emotion intensity prediction. For Track 2, they are fine-tuned for empathy and distress score prediction. The final prediction for each track is generated by averaging the outputs of these fine-tuned transformers, creating an ensemble that leverages the strengths of each individual model.
Datasets
The experimental datasets were provided by the WASSA 2023 shared task organizers.
- Track 1 (CONV):
- Training set: 8,776 conversations
- Validation set: 2,400 conversations
- Test set: 1,425 conversations
- Track 2 (EMP):
- Training set: 792 essays
- Validation set: 208 essays
- Test set: 100 essays
Key Insights and Ideas
- Ensemble Power: Averaging the predictions of multiple sentiment-enhanced RoBERTa models significantly boosts performance, demonstrating the effectiveness of ensemble methods in capturing nuanced aspects of empathy and emotion.
- Specialized Models Shine: RoBERTa-Twitter, despite being trained on tweets, showed strong performance on Track 1, suggesting its ability to generalize sentiment analysis to conversational contexts. EmoBERTa, designed for conversational emotion, also contributed positively.
- Track-Specific Strengths: While the ensemble approach generally improved results, individual models showed varying strengths across tracks. RoBERTa-Twitter was particularly strong for Track 1, while EmoBERTa showed promise for Track 2.
Key Results
The NCUEE-NLP system achieved impressive results in the WASSA 2023 shared task:
- Track 1 (Speech-turn level): The best submission achieved an average Pearson correlation of 0.7236, ranking fourth among all participating teams.
- Track 2 (Essay-level): The best submission obtained an average Pearson correlation of 0.4178, securing the first rank among all nine submissions.
Table 1 and Table 2 in the paper provide detailed performance metrics for individual models and the ensemble on both validation and test sets, showcasing the superiority of the ensemble approach.
Limitations and Future Work
The paper does not explicitly detail limitations or future work. However, the reliance on pre-trained transformer models and the ensemble approach suggests potential areas for future exploration:
- Further Model Specialization: Investigating even more specialized transformer architectures or fine-tuning strategies for specific types of emotional expression or empathy.
- Data Augmentation: Exploring data augmentation techniques to further enrich the training data, especially for the essay-level task.
- Interpretability: While the models achieve high performance, understanding why certain predictions are made remains a challenge in deep learning. Future work could focus on improving the interpretability of these empathy prediction models.
Summary updated