YNU-HPCC at WASSA-2023 Shared Task 1: Large-scale Language Model with LoRA Fine-Tuning for Empathy Detection and Emotion Classification

Yukun Wang, Jin Wang, Xuejie ZhangACL 2023

TL;DR

This paper presents a system for the WASSA-2023 Shared Task 1, focusing on predicting empathy, emotion, and personality from text. The team achieved strong results by fine-tuning large pre-trained language models (DeBERTa and RoBERTa) using parameter-efficient Low-Rank Adaptation (LoRA) and augmenting the dataset with back-translation.

What Problem Does This Paper Address?

The WASSA-2023 Shared Task 1 aims to predict nuanced human emotional and personality traits from text, specifically focusing on "empathic reactions." This involves understanding not just sentiment but also deeper psychological states like empathy, emotional intensity, personality traits (conscientiousness, openness, etc.), and interpersonal reactivity. Traditional NLP models often struggle with these complex, subjective aspects of human communication, requiring sophisticated methods to capture the underlying semantic and contextual information. The challenge lies in effectively learning these representations from limited task-specific data while managing the computational costs associated with large language models.

Methods

The YNU-HPCC team employed a strategy centered around leveraging powerful pre-trained language models (PLMs) and making their fine-tuning more efficient.

  • Pre-trained Models: They utilized two prominent PLMs:
  • DeBERTa (Decoding-enhanced BERT with disentangled attention): This model enhances BERT by incorporating disentangled attention mechanisms that consider word content and relative positions, along with an enhanced mask decoder. The XXL version (1.5B parameters) was used.
  • RoBERTa (Robustly optimized BERT pretraining approach): A refined version of BERT that removes the Next Sentence Prediction task, increases batch size, and uses more training data. The base version (125M parameters) was used.
  • Parameter-Efficient Fine-Tuning (PEFT) with LoRA: To mitigate the high computational cost of fine-tuning massive PLMs, they adopted Low-Rank Adaptation (LoRA). LoRA freezes most of the pre-trained model's parameters and injects trainable low-rank matrices into specific layers. This significantly reduces the number of trainable parameters (from 1.5B to around 4 million for XXL DeBERTa), making it feasible to train on consumer-grade GPUs.
  • Data Augmentation: Back-translation was used to augment the training dataset. This involves translating the original text into another language (Chinese in this case) and then translating it back into the original language (English). This process effectively doubles the training data, potentially improving model robustness and performance.
  • Task-Specific Heads: For classification tasks (like emotion classification), a softmax head was used. For regression tasks (predicting continuous values like empathy scores), a linear decoder head with mean squared error loss was employed.

Datasets

The task is based on an "Empathic Conversations" dataset. This dataset contains:

  • Conversations: Pairs of users reacting to news stories, labeled with emotional polarity, emotion, and empathy.
  • Essays: User-written essays, labeled with empathy, emotion, personality traits, and Interpersonal Reactivity Index (IRI).

The training dataset sizes are approximately 8700 for conversations and 770 for essays. The evaluation metrics used were Pearson Correlation for regression tasks (tracks 1, 2, 4, 5) and Macro F1-score for classification tasks (track 3).

Key Insights and Ideas

  • LoRA's Efficiency: The paper strongly demonstrates that LoRA can achieve performance comparable to full fine-tuning of large models, but with drastically reduced computational resources. This makes state-of-the-art NLP models more accessible for researchers and practitioners.
  • DeBERTa's Strength: The XXL version of DeBERTa, particularly with LoRA, showed superior performance on tracks 1, 3, and 5, highlighting its advanced text representation capabilities due to disentangled attention.
  • RoBERTa's Competitiveness: RoBERTa, while smaller, performed well on tracks 2 and 4, suggesting that model architecture and training data size can influence performance across different tasks.
  • Back-Translation Efficacy: The use of back-translation as a data augmentation technique proved effective, especially for tasks where training data might be limited.

Key Results

The system achieved competitive results on the WASSA-2023 Shared Task 1 leaderboard.

  • Development Set Performance:
    • DeBERTa+LoRA outperformed RoBERTa and BERT on tracks 1 (Empathy & Emotion in Conversations), 3 (Emotion Classification), and 5 (Interpersonal Reactivity Index).
  • RoBERTa showed better performance on tracks 2 (Empathy Prediction) and 4 (Personality Prediction).
  • Test Set Performance: Table 3 shows the final scores on the test dataset, with DeBERTa+LoRA achieving a Pearson Correlation of 0.730 for Track 1 and a Macro F1 of 0.514 for Track 3.

Limitations and Future Work

  • Performance Gap: The authors acknowledge that their system has room for improvement compared to top-ranked systems in the competition.
  • Future Directions: They plan to explore other text augmentation and generation methods to further enhance performance. This could involve more sophisticated techniques beyond simple back-translation.

Summary updated

BibTeX
@inproceedings{wang2023ynu,
  author = {Yukun Wang and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at WASSA-2023 Shared Task 1: Large-scale Language Model with LoRA Fine-Tuning for Empathy Detection and Emotion Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure