ConText at WASSA 2024 Empathy and Personality Shared Task: History-Dependent Embedding Utterance Representations for Empathy and Emotion Prediction in Conversations

Patrícia Pereira, Helena Moniz, Joao Paulo CarvalhoACL 2024

TL;DR

This paper introduces a novel approach for predicting empathy and emotion in conversations by leveraging history-dependent utterance representations. By feeding conversational context directly into pre-trained language models, they achieved top rankings in the WASSA 2024 shared task, demonstrating the power of contextualized embeddings for understanding nuanced human interaction.

What Problem Does This Paper Address?

Understanding and predicting empathy and emotion in conversations is crucial for building more effective and empathetic AI agents, with applications ranging from customer support to mental health. However, accurately capturing these subtle human traits is challenging because they are deeply context-dependent. Previous methods often processed each utterance independently before attempting to model their relationships, which can lose crucial historical information. This paper tackles the problem of how to best represent and utilize conversational history to improve empathy and emotion prediction.

Methods

The core of the proposed method lies in its "history-dependent embedding utterance representations." Instead of processing each utterance in isolation, the authors feed the target utterance along with a specified number of preceding turns directly into a pre-trained language model (PLM) like RoBERTa, DeBERTa, or Longformer. This allows the PLM to learn contextualized embeddings that inherently capture the conversational history.

For the CONV-turn track (predicting empathy, emotion polarity, and intensity of individual utterances), the input to the PLM is the current utterance preceded by a [CLS] token and followed by the previous c turns, separated by [SEP] tokens. A simple regression head is then appended to the PLM's output to predict the trait value.

For the CONV-dialog track (predicting perceived counterparty empathy), the approach is similar but includes all utterances from the conversation, ordered chronologically. Additionally, a special token indicating the interlocutor for whom empathy is being predicted is prepended to the input. This ensures the model can distinguish between the two participants in a dyadic conversation.

The authors experimented with different PLMs and varying numbers of context turns (c) to find the optimal configuration for each task.

Datasets

The research utilizes the dataset from the WASSA 2024 shared task on empathy and emotion prediction in interactions. This dataset consists of dyadic conversations in response to news articles depicting harm. The conversations are annotated at the turn level with perceived empathy, emotion polarity, and emotion intensity, and at the dialogue level with perceived counterparty empathy.

Key Insights and Ideas

  • Context is King: The results strongly emphasize that conversational history is not just beneficial but essential for accurately predicting empathy and emotion. Models with no context performed significantly worse.
  • Direct Context Integration: Feeding conversational history directly into PLMs, rather than post-hoc joint modeling, proves highly effective. This suggests that the Transformer architecture is adept at preserving and utilizing sequential information.
  • Task-Specific Context Needs: The optimal number of preceding turns (c) for context varies depending on the specific trait being predicted (e.g., 6 turns for Emotion Polarity, 2 for Emotion Intensity, and 4 for Empathy in the CONV-turn track).
  • Full Dialogue for Counterparty Empathy: For predicting perceived counterparty empathy, including the entire conversation history, rather than just a few turns, seems to provide the most comprehensive information.

Key Results

The proposed approach achieved significant success in the WASSA 2024 shared task:

  • CONV-turn Track: The system ranked 1st place for modeling empathy, emotion polarity, and emotion intensity.
  • CONV-dialog Track: The system ranked 2nd place for modeling perceived counterparty empathy, with a score very close to the top-ranked submission.

Specifically, on the test set for CONV-turn, their team achieved an average Pearson score of 0.626, outperforming other teams. For CONV-dialog, they achieved a Pearson score of 0.191, narrowly missing first place.

Limitations and Future Work

The authors acknowledge that while their approach for perceived counterparty empathy is promising, the achieved Pearson correlation score is still modest. They suggest that further validation on other datasets is needed to claim broader generalization.

For future work, they propose exploring different strategies for feeding context to the model, especially for the CONV-dialog task. Since the best-performing model (RoBERTa) has a token limit of 512, they suggest investigating whether using the final 512 tokens of a conversation, or other windowing strategies, might yield better results than the current approach of using all tokens (truncated if necessary).

Summary updated

BibTeX
@inproceedings{pereira2024context,
  author = {Patrícia Pereira and Helena Moniz and Joao Paulo Carvalho},
  title = {{ConText at WASSA 2024 Empathy and Personality Shared Task: History-Dependent Embedding Utterance Representations for Empathy and Emotion Prediction in Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure