hyy33 at WASSA 2024 Empathy and Personality Shared Task: Using the CombinedLoss and FGM for Enhancing BERT-based Models in Emotion and Empathy Prediction from Conversation Turns

Huiyu Yang, Liting Huang, Tian Li, Nicolay Rusnachenko, Huizhi LiangWassa 2024 · DOI

TL;DR

This paper tackles the challenge of predicting empathy, emotion polarity, and intensity in conversational turns, a crucial aspect of understanding human interaction. The authors propose a novel approach combining a structured contrastive loss with a Pearson correlation regularization term, enhanced by adversarial training with Fast Gradient Method (FGM) and data augmentation techniques. Their method achieved a strong 4th place in the WASSA 2024 shared task, demonstrating significant improvements after further refinements.

What Problem Does This Paper Address?

Understanding and accurately interpreting human emotions and empathy in conversations is a complex but vital task for numerous applications, from mental health support to improving human-computer interaction. The WASSA 2024 Shared Task on Empathy Detection and Emotion Classification and Personality Detection in Interactions specifically focuses on predicting perceived empathy, emotion polarity, and emotion intensity at the individual turn level within conversations. This is challenging because conversational nuances, subtle emotional shifts, and the subjective nature of empathy require models that can go beyond simple keyword matching and capture deeper contextual understanding. Existing methods often rely on standard BERT-based models, but the authors aim to push the boundaries by introducing novel loss functions and training strategies to improve robustness and accuracy in this specific domain.

Methods

The core of the proposed method involves fine-tuning pre-trained transformer models, specifically BERT and DeBERTa, on the conversational data. The key innovations lie in:

  • CombinedLoss: This custom loss function integrates a structured contrastive loss (for classification) with a Pearson correlation coefficient term. The Pearson correlation acts as a regularization, aiming to capture the subtle relationships between predicted and ground truth emotion levels, especially important for continuous or fine-grained labels.
  • Adversarial Training with Fast Gradient Method (FGM): To enhance model robustness and generalization, the authors employ adversarial training. FGM is used to compute perturbations that maximally increase the loss, and the model is then trained to minimize this perturbed loss. This helps the model become less sensitive to minor input variations.
  • Segmented Mix-up Augmentation: For further improvement, a novel data augmentation technique called "segmented mix-up" is introduced. This method mixes input samples and their labels within specific label ranges, preventing the generation of nonsensical intermediate samples that can occur with standard mix-up when dealing with highly contrasting labels.
  • Ensemble with Boosting: To build a more accurate and robust system, a boosting strategy is used to combine multiple fine-tuned models. Weights are assigned to models based on their performance on the development set, ensuring that more reliable predictions have a greater impact on the final output.

Datasets

The primary dataset used is from the WASSA 2024 Shared Task on Empathy Detection and Emotion Classification and Personality Detection in Interactions, specifically Track 2 (CONV-turn). This dataset includes:

  • Training Set: 11,166 samples.
  • Development Set: 990 samples.
  • Test Set: 2,061 samples.

Each sample consists of a dialogue turn's text content and corresponding labels for Emotion, Emotional Polarity, and Empathy. The dataset also includes meta-information like article IDs, conversation IDs, turn IDs, and speaker IDs. A notable point is that the development set contained emotion labels with float values (e.g., 0.5, 1.5) that were not present in the training set, which the authors identify as a limitation.

Key Insights and Ideas

  • DeBERTa Outperforms BERT: The DeBERTa model consistently shows superior performance compared to BERT, highlighting its enhanced capabilities for this specific task.
  • CombinedLoss is Crucial: The introduction of the CombinedLoss, particularly the Pearson correlation regularization, significantly improves performance by better capturing the nuances of emotion and empathy levels.
  • Adversarial Training Boosts Robustness: FGM-based adversarial training demonstrably improves the robustness and generalization ability of the models, leading to better overall performance.
  • Segmented Mix-up Prevents Nonsense: The segmented mix-up strategy is effective in generating meaningful augmented data, especially when dealing with a wide range of labels, leading to further performance gains.
  • Ensemble Benefits from Boosting: The boosting strategy for ensembling models proves effective, especially when combining models with varying strengths, leading to the highest overall scores.

Key Results

The paper reports several key quantitative results:

  • Initial Submission (DeBERTa + CombinedLoss + FGM): Achieved Pearson correlations of 0.581 for Emotion, 0.644 for Emotional Polarity, and 0.544 for Empathy on the test set, with an average of 0.590, ranking 4th in the competition.
  • Post-Submission Improvements (Development Set):
    • Fine-tuned DeBERTa with CombinedLoss and FGM achieved an average Pearson correlation of 0.6610.
  • With the addition of ensemble boosting and segmented mix-up, the model achieved even better results: 0.6521 for Emotion, 0.7376 for Emotional Polarity, and 0.6326 for Empathy.
  • Regression vs. Classification: The fine-tuned DeBERTa model performed slightly better in regression tasks compared to classification tasks, suggesting potential for future research.

Limitations and Future Work

The authors acknowledge two main limitations:

  1. Small Training Set: The training set size (less than 12,000 samples) is considered relatively small, and using larger datasets could potentially improve performance.
  2. Mismatched Labels: The inconsistency between training and development set labels (e.g., presence of float labels in development not seen in training) could lead to performance degradation if similar mismatches exist in the test set.

Future work directions include:

  • Larger Datasets: Retraining models on larger datasets, potentially at earlier stages like Masked Language Model pre-training, for better domain adaptation.
  • Incorporating More Context: Exploring the inclusion of conversational context and speaker personality for more comprehensive model construction.
  • Downstream Regression: Further investigating the performance of models in downstream regression tasks.

Summary updated

BibTeX
@inproceedings{yang2024hyy,
  author = {Huiyu Yang and Liting Huang and Tian Li and Nicolay Rusnachenko and Huizhi Liang},
  title = {{hyy33 at WASSA 2024 Empathy and Personality Shared Task: Using the CombinedLoss and FGM for Enhancing BERT-based Models in Emotion and Empathy Prediction from Conversation Turns}},
  booktitle = {Wassa},
  year = {2024},
  doi = {10.18653/v1/2024.wassa-1.39},
}
Figure