TL;DR
This paper presents a system for predicting empathy, emotion, and emotional polarity in conversations for the WASSA 2024 Shared Task. The researchers found that while advanced LLMs like GPT-3.5 Turbo showed promise, detailed prompting and careful context management were crucial for achieving competitive results, outperforming simpler models and even specialized single-task models.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically detecting and understanding empathy, emotion, and emotional polarity within conversational exchanges. Empathy, a complex human trait involving understanding and sharing another's feelings, is vital for effective social interaction and well-being. The WASSA 2024 Shared Task aims to develop computational models that can accurately predict these nuanced aspects of communication, particularly in the context of responses to news articles that describe harm. This is important for applications ranging from mental health support to building more emotionally intelligent AI systems.
Methods
The core of the system relies on fine-tuning a GPT-3.5 Turbo large language model (LLM). The researchers experimented with various prompting strategies, including simple instructions, detailed instructions, and the inclusion of few-shot examples. A key aspect of their approach was exploring the impact of conversational context, systematically varying the number of preceding turns provided to the LLM (from 2 to 10). They also investigated the utility of incorporating external linguistic features, specifically those derived from the LIWC (Linguistic Inquiry and Word Count) lexicon, and self-disclosure information. While initial experiments with LIWC features showed some correlation, they ultimately found that adding them decreased the LLM's performance, leading to their exclusion from the final system.
Datasets
The primary dataset used for this task is the one provided for the WASSA 2024 Shared Task 2 (Empathy and Emotion Prediction in Conversation Turns). This dataset includes conversations annotated with perceived empathy, emotion polarity, and emotion intensity. Additionally, the dataset contains demographic information (Big Five personality traits and Interpersonal Reactivity Index), though the authors found no significant correlation between this demographic data and the target scores, thus excluding it from their model. An external dataset, the Emotional Reactions Dataset, was also explored for augmentation, but it led to a significant performance drop, likely due to differing scoring systems.
Key Insights and Ideas
- Context is King: Providing conversational history significantly improves performance. The sweet spot for context was found to be around 5 previous turns, offering enough information without overwhelming the model.
- Prompting Matters Immensely: The way instructions are phrased to the LLM has a substantial impact. Detailed instructions explaining the task and the scoring criteria led to better results than simpler prompts.
- LLMs Don't Always Benefit from More Features: Surprisingly, adding LIWC features, which are often useful for sentiment analysis, actually degraded the performance of the GPT model in this specific empathy prediction task.
- Unified Prediction is Better: A single GPT model trained to predict all three scores (empathy, emotion polarity, and emotion) simultaneously performed better than specialized models trained to predict each score individually.
Key Results
The paper reports Pearson correlation coefficients as the primary evaluation metric. While specific final scores are not explicitly stated as the "best" in the main text, Table 4 shows that finetuning GPT-3.5 Turbo with 5 previous turns and detailed instructions yielded strong results, with an average Pearson coefficient around 0.7050 for the combined prediction. This approach outperformed classical machine learning models (RNN, LSTM, Bi-LSTM) and even a fine-tuned DeBERTaV2 model with LoRA, which achieved an average score of 0.6557. The authors note that their system achieved "relatively good results on the competition's official leaderboard."
Limitations and Future Work
The authors acknowledge that empathy and emotion prediction are inherently complex and challenging tasks, with limited research and annotated data available. They found that while LLMs are powerful, they did not significantly improve scores by simply adding more textual features. Future work could explore more sophisticated ways to integrate context and external knowledge, potentially addressing the limitations of differing scoring systems in external datasets. The paper also suggests that further research into the nuances of prompt engineering for LLMs in conversational AI tasks is warranted.
Summary updated