EmpatheticFIG at WASSA 2024 Empathy and Personality Shared Task: Predicting Empathy and Emotion in Conversations with Figurative Language

Gyeongeun Lee, Zhu Wang, Sathya N. Ravi, Natalie PardeWASSA 2024 · DOI · OpenReview

Abstract

This paper presents EmpatheticFIG, a framework developed for the WASSA 2024 shared task to predict empathy and emotion in conversations. By explicitly identifying and prompting Large Language Models (LLMs) with figurative language—specifically metaphors, idioms, and hyperboles—the system enhances the perception of emotional nuances. Our approach achieved 3rd and 4th place rankings in the shared tasks, demonstrating that linguistic cues significantly improve automated empathy detection.

Introduction

While Artificial Intelligence has made significant strides in sentiment analysis, "empathy"—the ability to understand and respond to the experiences of others—remains a challenging frontier in natural language processing, particularly in back-and-forth dialogue. A key missing ingredient in automated empathy detection is the nuanced interpretation of figurative language. Humans frequently employ metaphors ("I'm drowning in work"), idioms ("pull through the hard times"), and hyperbole ("thousands of years of progress") to amplify emotional impact. Standard models often treat these expressions literally, missing their underlying emotional "charge." This research explores whether explicitly informing a model about the presence of figurative language helps it better perceive underlying empathy and emotion.

Methodology

The proposed EmpatheticFIG framework utilizes a multi-stage approach to integrate figurative language awareness into emotion prediction:

  1. Figurative Language Detection: A pre-trained mT5 model scans conversation texts to identify metaphors, idioms, and hyperboles.
  2. Prompt Augmentation: The raw text is augmented with a descriptive prompt indicating the detected figurative types (e.g., "The text contains metaphor, hyperbole").
  3. Encoding: These augmented strings are processed by transformer-based encoders, including BERT, DeBERTa, RoBERTa, T5, and Llama-3-8b. For the Llama-3 model, Parameter-Efficient Fine-Tuning (PEFT) was employed.
  4. Task-Specific Prediction: A Multilayer Perceptron (MLP) layer outputs scores for perceived empathy, emotion polarity, and emotion intensity.
The EmpatheticFIG framework architecture.

Datasets

The study utilized the Empathic Conversations dataset provided for the WASSA 2024 shared task, divided into two tracks:

  • Track 1 (CONV-dialog): Predicting perceived empathy at the overall dialogue level (1,037 texts).
  • Track 2 (CONV-turn): Predicting empathy, emotion polarity, and intensity at the individual speech-turn level (12,080 texts).

Analysis revealed that figurative language is highly prevalent, appearing in approximately 91% of Track 1 samples and 80% of Track 2 samples.

Results and Discussion

The EmpatheticFIG system proved highly competitive, ranking 3rd in Track 2 and 4th in Track 1 officially. Key insights from the experiments include:

  • Feature Specificity: Different figurative types benefit different tasks. Metaphors and hyperboles were most effective for dialogue-level empathy, while metaphors specifically helped gauge emotion intensity.
  • The Idiom Paradox: Interestingly, adding idiom features occasionally decreased performance. This may be due to the high baseline prevalence of idioms, which might introduce noise if the model already possesses a baseline understanding of common idiomatic expressions.
  • Contextual Impact: Figurative prompts had a more significant impact on dialogue-level empathy (Track 1) than on turn-level predictions, suggesting these cues help models synthesize the overall emotional "vibe" of a conversation.
Performance of DeBERTa with various figurative feature combinations on Track 2.

Conclusion

EmpatheticFIG demonstrates that explicit awareness of figurative language can significantly enhance an LLM's ability to perceive empathy and emotion. Despite strong development set performance, a drop in test set results for Track 1 suggests challenges with overfitting on smaller, imbalanced datasets. Future work will focus on scaling to larger LLMs, refining detection accuracy, and exploring additional rhetorical devices such as irony and personification.

Summary updated

Abstract

Recent research highlights the importance of figurative language as a tool for amplifying emotional impact. In this paper, we dive deeper into this phenomenon and outline our methods for Track 1, Empathy Prediction in Conversations (CONV-dialog) and Track 2, Empathy and Emotion Prediction in Conversation Turns (CONV-turn) of the WASSA 2024 shared task. We leveraged transformer-based large language models augmented with figurative language prompts, specifically idioms, metaphors and hyperbole, that were selected and trained for each track to optimize system performance. For Track 1, we observed that a fine-tuned BERT with metaphor and hyperbole features outperformed other models on the development set. For Track 2, DeBERTa, with different combinations of figurative language prompts, performed well for different prediction tasks. Our method provides a novel framework for understanding how figurative language influences emotional perception in conversational contexts. Our system officially ranked 4th in the 1st track and 3rd in the 2nd track.
BibTeX
@inproceedings{lee2024empatheticfig,
  author = {Gyeongeun Lee and Zhu Wang and Sathya N. Ravi and Natalie Parde},
  title = {{EmpatheticFIG at WASSA 2024 Empathy and Personality Shared Task: Predicting Empathy and Emotion in Conversations with Figurative Language}},
  booktitle = {WASSA},
  year = {2024},
  doi = {10.18653/v1/2024.wassa-1.41},
}
Figure