Abstract
This paper presents EmpatheticFIG, a framework developed for the WASSA 2024 shared task to predict empathy and emotion in conversations. By explicitly identifying and prompting Large Language Models (LLMs) with figurative language—specifically metaphors, idioms, and hyperboles—the system enhances the perception of emotional nuances. Our approach achieved 3rd and 4th place rankings in the shared tasks, demonstrating that linguistic cues significantly improve automated empathy detection.
Introduction
While Artificial Intelligence has made significant strides in sentiment analysis, "empathy"—the ability to understand and respond to the experiences of others—remains a challenging frontier in natural language processing, particularly in back-and-forth dialogue. A key missing ingredient in automated empathy detection is the nuanced interpretation of figurative language. Humans frequently employ metaphors ("I'm drowning in work"), idioms ("pull through the hard times"), and hyperbole ("thousands of years of progress") to amplify emotional impact. Standard models often treat these expressions literally, missing their underlying emotional "charge." This research explores whether explicitly informing a model about the presence of figurative language helps it better perceive underlying empathy and emotion.
Methodology
The proposed EmpatheticFIG framework utilizes a multi-stage approach to integrate figurative language awareness into emotion prediction:
- Figurative Language Detection: A pre-trained mT5 model scans conversation texts to identify metaphors, idioms, and hyperboles.
- Prompt Augmentation: The raw text is augmented with a descriptive prompt indicating the detected figurative types (e.g., "The text contains metaphor, hyperbole").
- Encoding: These augmented strings are processed by transformer-based encoders, including BERT, DeBERTa, RoBERTa, T5, and Llama-3-8b. For the Llama-3 model, Parameter-Efficient Fine-Tuning (PEFT) was employed.
- Task-Specific Prediction: A Multilayer Perceptron (MLP) layer outputs scores for perceived empathy, emotion polarity, and emotion intensity.
Datasets
The study utilized the Empathic Conversations dataset provided for the WASSA 2024 shared task, divided into two tracks:
- Track 1 (CONV-dialog): Predicting perceived empathy at the overall dialogue level (1,037 texts).
- Track 2 (CONV-turn): Predicting empathy, emotion polarity, and intensity at the individual speech-turn level (12,080 texts).
Analysis revealed that figurative language is highly prevalent, appearing in approximately 91% of Track 1 samples and 80% of Track 2 samples.
Results and Discussion
The EmpatheticFIG system proved highly competitive, ranking 3rd in Track 2 and 4th in Track 1 officially. Key insights from the experiments include:
- Feature Specificity: Different figurative types benefit different tasks. Metaphors and hyperboles were most effective for dialogue-level empathy, while metaphors specifically helped gauge emotion intensity.
- The Idiom Paradox: Interestingly, adding idiom features occasionally decreased performance. This may be due to the high baseline prevalence of idioms, which might introduce noise if the model already possesses a baseline understanding of common idiomatic expressions.
- Contextual Impact: Figurative prompts had a more significant impact on dialogue-level empathy (Track 1) than on turn-level predictions, suggesting these cues help models synthesize the overall emotional "vibe" of a conversation.
Conclusion
EmpatheticFIG demonstrates that explicit awareness of figurative language can significantly enhance an LLM's ability to perceive empathy and emotion. Despite strong development set performance, a drop in test set results for Track 1 suggests challenges with overfitting on smaller, imbalanced datasets. Future work will focus on scaling to larger LLMs, refining detection accuracy, and exploring additional rhetorical devices such as irony and personification.
Summary updated