Chinchunmei at WASSA 2024 Empathy and Personality Shared Task: Boosting LLM's Prediction with Role-play Augmentation and Contrastive Reasoning Calibration

Tian Li, Nicolay Rusnachenko, Huizhi LiangWassa 2024 · DOI

TL;DR

This paper tackles the challenge of accurately detecting empathy and personality traits in text using large language models (LLMs). The Chinchunmei team developed novel fine-tuning strategies, including role-play augmentation and contrastive reasoning calibration, to significantly improve LLM performance on the WASSA 2024 shared task, achieving top rankings across multiple tracks.

What Problem Does This Paper Address?

Empathy, the ability to understand and share others' feelings, is a crucial human trait with implications across various fields. Accurately identifying and quantifying empathy in text, especially within diverse contexts like dialogues, articles, and essays, remains a significant challenge. The WASSA 2024 shared task aimed to address this by introducing multi-level and multi-modal data, moving beyond simple classification to a scoring system. A key difficulty lies in the inherent nuances of human expression, the limited availability of training data for specific tasks, and the potential for LLMs, primarily trained for next-token prediction, to struggle with discriminative scoring tasks. This paper focuses on enhancing LLM capabilities for this complex empathy detection task.

Methods

The Chinchunmei team employed a two-stage approach using the Llama3-8b-instruct LLM.

1. Supervised Fine-Tuning (SFT) Stage: * Standard Prediction: A baseline fine-tuning approach where the LLM predicts scores directly. * Role-play Template: To better capture the dynamics of dialogues, the model was fine-tuned to generate responses from a specific role's perspective, enhancing its understanding of inter-role empathy. * Contrastive Supervised Fine-Tuning (C-SFT): To combat data scarcity and improve the model's ability to understand comparable label values, they created contrastive pairs of data points. This forces the model to learn by comparing predictions between similar but distinct examples, rather than treating each prediction in isolation.

2. Inference Stage: * Contrastive Reasoning Calibration (CRC): This technique leverages Chain-of-Thought (CoT) during inference. The model compares predictions for a given label between two data points, using the contrastive training data as benchmarks. This process refines the final predictions, making them more reliable and discriminative.

Datasets

The primary datasets used were from the WASSA 2024 Shared Task 1: Empathy Detection and Emotion Classification. This task involved:

  • Track 1: Dialogue data for empathy detection.
  • Track 2: Dialogue data for emotion intensity, empathy, and emotion polarity.
  • Track 3: Article and essay data for empathy and distress prediction.

The paper notes that Tracks 1 and 3 had limited training samples (around 1,000 each), highlighting the importance of their data augmentation techniques.

Key Insights and Ideas

  • Role-play Augmentation is Key for Dialogues: Fine-tuning LLMs with role-playing scenarios significantly improved their ability to understand and predict empathy in conversational contexts.
  • Contrastive Learning for Data Scarcity: C-SFT proved effective in overcoming limited training data by teaching the model to discern subtle differences between examples through pairwise comparisons.
  • Inference-Time Calibration Matters: CRC, by incorporating contrastive reasoning at inference, further boosted the accuracy and reliability of the LLM's scoring, demonstrating that post-training calibration can be highly impactful.
  • LLMs are Powerful for Empathy Detection: The success across multiple tracks underscores the potential of LLMs, even with modest fine-tuning, for complex subjective tasks like empathy detection.

Key Results

The Chinchunmei team achieved strong results across the WASSA 2024 shared task:

  • Track 1 (Empathy Detection): Achieved 0.17 on the test set.
  • Track 2 (Emotion, Empathy, Polarity): Achieved 0.61 (Emotion), 0.68 (Empathy), and 0.58 (Polarity) on the test set.
  • Track 3 (Empathy, Distress): Achieved 0.33 (Empathy) and 0.35 (Distress) on the test set.

Notably, their approach secured third place in Track 1 and second place in Tracks 2 and 3. The paper also shows that increasing the number of contrastive samples (i) during CRC generally led to performance improvements on Track 2, though this was constrained by computational resources during the competition.

Limitations and Future Work

  • Track 3 Performance Decline: The authors observed a performance decline in Track 3 after using C-SFT. They hypothesize this might be due to the loss of article content information, which is crucial for the complex labeling system in Track 3. They plan to investigate this further with long-context LLMs.
  • Computational Constraints: The choice of i=1 for CRC during the competition was a resource-driven decision. Further experiments with larger i values showed potential for improvement, suggesting that more extensive contrastive reasoning at inference could yield even better results.
  • Generalizability: While the techniques are presented as transferable, their effectiveness on other similar tasks would require further validation. The authors suggest their methods can be "easily transferred to other similar tasks."

Summary updated

BibTeX
@inproceedings{li2024chinchunmei,
  author = {Tian Li and Nicolay Rusnachenko and Huizhi Liang},
  title = {{Chinchunmei at WASSA 2024 Empathy and Personality Shared Task: Boosting LLM's Prediction with Role-play Augmentation and Contrastive Reasoning Calibration}},
  booktitle = {Wassa},
  year = {2024},
  doi = {10.18653/v1/2024.wassa-1.32},
}
Figure