Zhenmei at WASSA-2024 Empathy and Personality Shared Track 2 Incorporating Pearson Correlation Coefficient as a Regularization Term for Enhanced Empathy and Emotion Prediction in Conversational Turns

Liting Huang, Huizhi LiangWassa 2024 · DOI

TL;DR

This paper introduces a novel regularization technique using the Pearson correlation coefficient to improve emotion and empathy prediction in conversational AI. By encouraging predicted emotions to align linearly with true emotions, their method leads to smoother, more accurate predictions and achieved a 9th place ranking in the WASSA-2024 CONV-turn track.

What Problem Does This Paper Address?

Conversational AI systems strive to understand and respond to human emotions, which are complex and often exist on a continuous spectrum rather than discrete categories. While many models focus on minimizing prediction errors, they often overlook the linear relationship between predicted and actual emotional states. This can lead to "extreme" or unstable predictions, failing to capture the nuanced transitions in human emotion. The authors aim to address this by ensuring that not only are predictions accurate, but they also maintain a strong linear correlation with the ground truth, leading to more human-like and consistent emotional understanding.

Methods

The core innovation is the incorporation of the Pearson correlation coefficient as a regularization term within the loss function. Instead of solely minimizing prediction errors (like Cross-Entropy Loss), the model is also penalized for a low Pearson correlation between its predictions ($\hat{y}$) and the true labels ($y$). This is achieved by adding a term like $\lambda \cdot (1 - \text{Pearson}(\hat{y}, y))$ to the total loss, where $\lambda$ is a hyperparameter controlling the regularization's strength.

The authors also employ two additional techniques:

  • Consistent-Mixup Data Augmentation (CMDA): This method creates new training samples by interpolating between existing ones, but crucially, it ensures that the labels of the interpolated samples remain consistent. This helps the model generalize better without introducing label noise.
  • Boosting: They combine the predictions of multiple strong models (BERT, RoBERTa, DeBERTa) using a weighted average, leveraging their individual strengths to achieve superior overall performance.

Datasets

The primary dataset used is the Track 2 (CONV-turn) dataset. This dataset contains conversational texts with three associated labels: emotional intensity, emotional polarity, and empathy. The dataset has 11,166 training items, 990 development items, and approximately 2,300 test items. The authors mention that the data was preprocessed to identify and merge redundant columns, ensuring alignment with the features.

Key Insights and Ideas

  • Correlation as a Regularizer is Key: The most significant insight is that simply minimizing prediction error isn't enough for nuanced emotion prediction. Ensuring a high linear correlation between predicted and actual emotions is crucial for generating stable and human-like outputs.
  • Beyond Discrete Labels: The paper highlights the limitations of treating emotions as purely discrete categories. The Pearson correlation approach inherently handles the continuous nature of emotions, leading to better performance on tasks that require understanding subtle emotional shifts.
  • Synergy of Techniques: The combination of Pearson regularization, CMDA, and boosting proved to be more effective than any single technique, demonstrating the power of a multi-faceted approach to model enhancement.

Key Results

The paper reports significant performance improvements across various configurations.

  • Pearson Regularization Boosts Performance: Models enhanced with Pearson regularization consistently outperformed their baseline counterparts on Emotion, Emotional Polarity, and Empathy tasks.
  • Boosting Further Enhances Results: The Boosting (Standard) model showed improvement over individual models. The Boosting (Pearson) model, which combined Pearson-regularized models, achieved even higher scores.
  • Top Performance with Combined Approach: The Boosting (Pearson, CMDA) model achieved the best results, with top scores in Emotional Polarity (0.765) and Empathy (0.658) on the development set.
  • Competitive Ranking: The system officially ranked 9th at the WASSA-2024 Track 2: CONV-turn.
  • Test Set Improvements: The Boosting (Pearson, CMDA) model achieved scores of 0.581 (Emotion), 0.644 (Emotional Polarity), and 0.561 (Empathy) on the test set. The authors note these are significantly higher than an earlier submission due to a mistake.

Limitations and Future Work

  • Early Submission Issue: The authors acknowledge that an earlier version of their model was submitted by mistake, leading to lower scores on the test set. The results presented in the paper reflect the final, optimized models.
  • Limited Metrics: Relying solely on the Pearson Correlation Coefficient might not fully capture the model's effectiveness. The authors suggest including other metrics, such as the F1 score, in future work for a more comprehensive evaluation.
  • Time Constraints: Due to time constraints, they were unable to explore all potential avenues, leading to the early submission issue. Future work could involve more extensive experimentation and metric selection.

Summary updated

BibTeX
@inproceedings{huang2024zhenmei,
  author = {Liting Huang and Huizhi Liang},
  title = {{Zhenmei at WASSA-2024 Empathy and Personality Shared Track 2 Incorporating Pearson Correlation Coefficient as a Regularization Term for Enhanced Empathy and Emotion Prediction in Conversational Turns}},
  booktitle = {Wassa},
  year = {2024},
  doi = {10.18653/v1/2024.wassa-1.34},
}
Figure