Empaths at WASSA 2024 Empathy and Personality Shared Task: Turn-Level Empathy Prediction Using Psychological Indicators

Shaz Furniturewala, Kokil JaidkaACL 2024

TL;DR

This paper introduces a novel approach to predicting empathy in conversations by decomposing it into six psychological indicators. By enriching text data with these indicators, generated by a large language model (LLM), and then fine-tuning a transformer model, the researchers significantly improved empathy prediction accuracy, achieving a 7th place ranking in the WASSA 2024 shared task. This work highlights the importance of understanding the underlying psychological constructs of empathy for more robust and interpretable AI systems.

What Problem Does This Paper Address?

Accurately detecting empathy in human-computer interactions is crucial for developing more responsive and supportive AI systems, from customer service bots to therapeutic agents. Traditional methods in Natural Language Processing (NLP) often rely on surface-level features like sentiment analysis and keyword matching. While these methods can capture some aspects of empathy, they struggle with its nuanced, context-dependent nature, which is deeply rooted in complex psychological constructs. The challenge lies in moving beyond simple textual cues to capture the deeper cognitive and affective dimensions that constitute genuine empathy. This paper aims to bridge this gap by explicitly modeling these underlying psychological indicators.

Methods

The core of this research is a two-stage approach:

  1. Psychological Indicator Enrichment: The authors leverage GPT-40, a powerful LLM, to analyze each utterance in the dataset and assess six predefined psychological indicators: Emotional Language, Perspective-Taking, Sympathy and Compassion, Extroversion, Openness, and Agreeableness. For each indicator, GPT-40 provides a rating (Low, Medium, High) and a concise, single-sentence explanation, highlighting the textual evidence. This enriched information is then concatenated with the original utterance.
  2. DeBERTa Fine-tuning: A DeBERTa V3 classifier, known for its strong performance in NLP tasks due to its advanced attention mechanisms, is then fine-tuned on this enriched dataset. The model learns to predict empathy levels by considering both the original text and the LLM-generated psychological insights.

The researchers also explored zero-shot classification with GPT-40 itself, but found that fine-tuning a dedicated transformer model yielded superior results.

Datasets

The study utilizes the dataset created for the WASSA 2024 Empathy and Personality Shared Task. This dataset consists of approximately 500 conversations between Amazon Mechanical Turk (AMT) workers reacting to news articles about negative events. Each conversation has over 15 turns, and the data is annotated at the turn-level with empathy scores ranging from 0 to 5. The authors created their own test set from a subset of the provided training data for their unofficial results.

Key Insights and Ideas

  • Decomposition is Key: Breaking down the complex construct of empathy into distinct psychological indicators (like perspective-taking and sympathy) provides a more granular and interpretable framework for AI models.
  • LLMs as Psychological Annotators: LLMs like GPT-40 can be effectively employed to generate nuanced, human-interpretable annotations for psychological constructs, going beyond simple sentiment.
  • Transformer Models Benefit from Context: Fine-tuning transformer models like DeBERTa on text enriched with psychological indicators significantly boosts their empathy prediction capabilities, demonstrating the value of explicit contextual information.
  • Extroversion's Counter-Intuitive Role: The study found a negative correlation between Extroversion and empathy in this dataset, suggesting that outward sociability might not always align with empathetic responses in certain conversational contexts.
  • LLMs Still Have Room to Grow: While GPT-40's explanations are valuable, its zero-shot performance for empathy prediction was modest, indicating that even advanced LLMs may require further fine-tuning or different reasoning-based approaches to fully grasp complex human emotions in dialogue.

Key Results

The DeBERTa model, when augmented with the psychological indicators, showed significant improvements:

  • Pearson Correlation Coefficient: Increased from 0.65 (utterance only) to 0.68 (utterance + indicators).
  • F1 Score: Improved from 0.32 to 0.35.
  • Accuracy: Rose from 0.52 to 0.55.

The zero-shot GPT-40 approach saw a smaller improvement in Pearson correlation (0.38 to 0.41) but a slight decrease in F1 score and accuracy, suggesting it could benefit from further fine-tuning. The authors' system achieved an unofficial 7th place ranking on the CONV-turn track of the WASSA 2024 shared task.

Limitations and Future Work

The authors acknowledge a primary limitation: their reliance on GPT-40 for data enrichment. This dependence means the interpretation of labels is sensitive to prompt design, potentially leading to "concept drift." To address this, future work will involve obtaining additional expert annotations and conducting a thorough inspection of GPT-40's reasoning to ensure the validity and nuance of the enriched data. They also note that their reported results are unofficial, based on a subset of the data, and they were unable to test on the official test set.

Summary updated

BibTeX
@inproceedings{furniturewala2024empaths,
  author = {Shaz Furniturewala and Kokil Jaidka},
  title = {{Empaths at WASSA 2024 Empathy and Personality Shared Task: Turn-Level Empathy Prediction Using Psychological Indicators}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure