TL;DR
This paper presents PICT-CLRL's approach to the WASSA 2023 Empathy, Emotion, and Personality Shared Task, focusing on predicting empathy and distress levels in essays. They achieved third place by employing an ensemble of transformer-based models, demonstrating the effectiveness of combining predictions from models like RoBERTa and SimCSE.
What Problem Does This Paper Address?
Understanding and quantifying human emotions like empathy and distress from text is a significant challenge in Natural Language Processing (NLP). While sentiment analysis has seen considerable progress, empathy and distress detection are more nuanced and crucial for comprehending human interactions, fostering social connections, and identifying potential harm or suffering. The WASSA 2023 shared task specifically aimed to predict empathy and distress levels from essays written in response to news stories about people facing hardships. This requires models to go beyond surface-level sentiment and grasp deeper emotional states, often within longer text formats.
Methods
The PICT-CLRL team experimented with several transformer-based models, leveraging the power of pre-trained language models. Their approach involved:
- Individual BERT-based Models: They evaluated pre-trained models such as RoBERTa (Liu et al., 2019), which benefits from extensive pre-training on a large corpus, and Twitter-RoBERTa-emotion and Twitter-RoBERTa-sentiment, models specifically pre-trained on Twitter data. They also incorporated the unsupervised SimCSE (Gao et al., 2022) model, which uses sentence embeddings instead of token embeddings, showing promise for longer texts.
- Ensemble Methods: To further improve performance, they employed ensemble techniques. This involved combining the predictions of multiple individual models. They explored simple averaging of predictions and more sophisticated machine learning approaches like linear regression, Support Vector Regression (SVR), and XGBoost. The core idea behind ensembling is that combining diverse model outputs can lead to more robust and accurate predictions than any single model alone.
Datasets
The primary dataset used for this task was the Empathic Conversations dataset presented by Omitaomu et al. (2022). This dataset consists of essays, ranging from 300 to 800 characters, written in response to news stories about individuals experiencing difficulties. Each essay is associated with numerical scores for empathy and distress. The dataset also includes various demographic attributes (age, gender, education) of the essay writers. The authors report the dataset splits as 792 examples for training, 208 for validation, and 100 for testing.
Key Insights and Ideas
- Sentence Embeddings Shine for Long Texts: The unsupervised SimCSE model, which utilizes sentence embeddings, demonstrated superior performance in predicting distress. This suggests that for longer text documents, focusing on sentence-level representations can be more effective than traditional token-level embeddings.
- Ensembling is Key to Robustness: Averaging the predictions of individual models proved to be the most effective ensembling strategy, yielding the best overall results. This highlights the power of simple aggregation in mitigating individual model weaknesses and producing stable predictions.
- Domain-Specific Pre-training Isn't Always Better: While Twitter-specific RoBERTa models were tested, they did not significantly outperform the general RoBERTa-base model. This might indicate that the casual nature of tweets differs substantially from the more formal and reflective style of the essays in the dataset.
- Demographic Information's Role: Although not explicitly used in their final model, the paper acknowledges that demographic information (like age, gender, education) is provided in the dataset and can play a role in empathy detection, as explored by other works (e.g., Guda et al., 2021).
Key Results
The PICT-CLRL team achieved a Pearson's r score of 0.346 for their final submission, securing third place in the Empathy Prediction (EMP) track of the WASSA 2023 shared task. Individually, the unsupervised SimCSE model achieved the highest average Pearson correlation of 0.352, outperforming other models. Among the ensemble methods, simple averaging of model predictions resulted in the best overall performance.
Limitations and Future Work
- Computational Cost: The authors acknowledge that while their transformer-based models perform well on the provided dataset, their computational requirements might limit their feasibility in real-world scenarios. Developing more efficient models is suggested as a future direction.
- Data Size: The relatively small size of the test dataset (100 samples) might have impacted the performance of more complex ensembling methods like SVR and XGBoost, as they struggled to fully utilize the information.
- Further Exploration: The paper suggests exploring additional pre-trained models, particularly those trained on data more relevant to the task's domain, and investigating other ensembling techniques to further enhance performance.
Summary updated