TL;DR
This paper details the IITP-AINLPML team's participation in the WASSA 2022 shared task, focusing on empathy detection, emotion classification, and personality prediction. They developed multi-task deep learning models for empathy/emotion tasks and SVMs leveraging demographic data for personality/reactivity index prediction, achieving competitive results across all tracks.
What Problem Does This Paper Address?
The paper tackles the challenge of computationally understanding human emotions and personality traits expressed in text, a crucial step for improving human-computer interaction. While prior work exists, the authors highlight shortcomings like simplistic definitions of empathy and a lack of publicly available corpora. The WASSA 2022 Shared Task aimed to advance this field by introducing four tracks: empathy and distress prediction, emotion classification, personality prediction, and interpersonal reactivity index prediction, all based on news story essays.
Methods
The team employed a multi-task learning approach for the empathy and emotion classification tasks. For empathy and distress prediction (Track 1), they fine-tuned a pre-trained BERT encoder on essays, using a shared dense layer followed by task-specific layers. For emotion classification (Track 2), they further leveraged transfer learning by first training a BERT encoder on an augmented dataset and then fine-tuning another BERT encoder on the WASSA 2022 emotion data.
For personality (PER) and Interpersonal Reactivity Index (IRI) prediction (Tracks 3 & 4), they found that using essay text with deep learning yielded poor results. Instead, they opted for Support Vector Machines (SVMs) trained on demographic information (gender, race, education, age, income) associated with the authors. Specific demographic features were chosen based on their observed correlation with particular personality traits.
Datasets
The primary dataset used was the WASSA 2022 Shared Task dataset, which consists of news story essays. For augmenting the emotion classification task, they utilized three publicly available datasets: ISEAR, Crowdflower's Text Emotion dataset, and SemEval 2018 Task 1 English Emotion Classification dataset.
Key Insights and Ideas
- Multi-task learning is beneficial: Jointly learning correlated tasks like empathy and distress, or emotion with empathy/distress, significantly improved overall system performance. This suggests that these emotional and empathetic constructs share underlying linguistic signals.
- Demographics matter for personality: Essay text alone was insufficient for predicting personality traits and reactivity indices. Incorporating demographic information proved to be a more effective strategy for these specific tasks.
- Transfer learning is crucial for limited data: The small size of the WASSA 2022 training set for emotion classification necessitated a transfer learning approach, which proved beneficial.
- Test set distribution shift: The authors observed unexpectedly low scores on the test set compared to the development set for tracks 2, 3, and 4, hypothesizing a potential distribution shift in the test data.
Key Results
The IITP-AINLPML team achieved competitive rankings in the WASSA 2022 Shared Task:
- Track 1 (Empathy): Average Pearson score of 0.483 on the test set, ranking 8th.
- Track 2 (Emotion Classification): Macro F1 score of 0.524 on the test set, ranking 11th.
- Track 3 (Personality): Average Pearson score of 0.05 on the test set, ranking 2nd.
- Track 4 (Interpersonal Reactivity Index): Average Pearson score of 0.08 on the test set, ranking 2nd.
Their multi-task systems consistently outperformed single-task systems.
Limitations and Future Work
The authors acknowledge the small size of the released training data as a primary limitation affecting performance across all tracks. For emotion classification, data imbalance across classes also led to biased predictions.
Future work includes:
- Incorporating lexicon features from available lexical resources to enhance EMP and EMO tasks.
- Developing techniques to combine contextual information from essays with demographic data for more robust PER and IRI predictions.
- Investigating the observed distribution shift between development and test sets.
Summary updated