RU at WASSA 2024 Shared Task: Task-Aligned Prompt for Predicting Empathy and Distress

Haein Kong, Seonghyeon MoonACL 2024

TL;DR

This paper introduces a "Task-Aligned Prompt" (TAP) strategy to improve the ability of Large Language Models (LLMs) to predict empathy and distress from text. By explicitly defining empathy and distress according to the task's specific context and providing relevant background information, their approach significantly boosted LLM performance, leading to a top-ranked submission in the WASSA 2024 Shared Task.

What Problem Does This Paper Address?

Predicting psychological states like empathy and distress from text is a crucial task with applications in mental health and human-computer interaction. While Large Language Models (LLMs) have shown promise in this area, their performance can be inconsistent. This inconsistency stems from the fact that LLMs might interpret abstract psychological constructs differently than how they are defined within a specific research task. Furthermore, the way individuals express empathy and distress can vary greatly depending on the context. This paper addresses the challenge of aligning LLM understanding with the precise definitions and contextual nuances required for accurate empathy and distress prediction in a specific task setting.

Methods

The core contribution of this paper is the Task-Aligned Prompt (TAP) strategy. This approach aims to bridge the gap between an LLM's general understanding and the specific requirements of the empathy and distress prediction task. TAP has two main components:

  1. Definition Alignment: Instead of relying on the LLM's default understanding of "empathy" and "distress," TAP explicitly provides the definitions as established in the original research that collected the dataset (Buechel et al., 2018) and the measurement scales used (Batson et al., 1987). This ensures the LLM is working with the task-specific definitions.
  2. Dataset Alignment (Contextual Information): The prompt also includes contextual information about the dataset. For this specific task, this meant informing the LLM that the text data consists of people's reactions to news articles, shared as if in a private message or social media post. This helps the LLM understand the nature of the communication and how empathy/distress might be expressed.

These TAP prompts were tested using ChatGPT and GPT-40 in both zero-shot (no examples) and few-shot (a few examples) settings. The performance was compared against "plain prompts" that only included the task instruction and output format.

Datasets

The study utilized the dataset from Track 3: Empathy Prediction (EMP) of the WASSA 2024 Shared Task. This dataset consists of:

  • Essays written by individuals.
  • Scores for empathy and distress, measured on a 7-Likert scale using the Batson's Empathic Concern – Personal Distress Scale. Scores range from 1 (not at all) to 7 (extremely).

The dataset was divided into:

  • Train: 1000 instances (used for few-shot examples).
  • Dev: 63 instances (used for development set experiments).
  • Test: 83 instances (used for final evaluation).

No new benchmarks were introduced; the paper focused on applying LLMs to an existing benchmark.

Key Insights and Ideas

  • LLM Definitions Matter: The paper strongly suggests that LLMs don't inherently "know" the precise, nuanced definitions of psychological constructs as used in specific research tasks. Providing these definitions directly in the prompt is crucial for performance.
  • Context is King: Simply providing definitions isn't enough. Informing the LLM about the context in which the text was generated (e.g., reactions to news articles shared socially) significantly aids in interpreting how empathy and distress are expressed.
  • Zero-Shot Power: The TAP strategy proved particularly effective in zero-shot settings, demonstrating that well-crafted prompts can elicit strong performance from LLMs without the need for extensive task-specific fine-tuning or example provision.
  • TAP Outperforms Plain Prompts: Across most tested scenarios, the Task-Aligned Prompt consistently outperformed plain prompts, highlighting the efficacy of the proposed alignment strategy.

Key Results

The experimental results on the development set showed that the TAP-ChatGPT-zero-shot model achieved the highest average Pearson correlation for empathy and distress (r = .51). While plain prompts performed slightly better for empathy prediction with ChatGPT zero-shot (r = .49), TAP generally led to better overall performance.

For the final evaluation on the EMP track, the TAP-ChatGPT-zero-shot model was used. Their team, RU, achieved the top rank with an average Pearson coefficient of 0.453. Specifically, their submission achieved a Pearson coefficient of 0.523 for empathy and 0.383 for distress.

Limitations and Future Work

The authors acknowledge that their approach's generalizability might be limited. Because TAP emphasizes aligning detailed nuances and contexts to specific tasks, it may not be the most effective method for predicting empathy and distress across diverse datasets with different contexts and backgrounds.

Future research directions suggested include:

  • Developing more generalizable prompting strategies that can be applied across multiple datasets.
  • Improving the interpretability of LLM predictions for empathy and distress.
  • Exploring ways to add more layers to enhance the explainability and interpretability of LLMs in this domain.

Summary updated

BibTeX
@inproceedings{kong2024ru,
  author = {Haein Kong and Seonghyeon Moon},
  title = {{RU at WASSA 2024 Shared Task: Task-Aligned Prompt for Predicting Empathy and Distress}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure