YNU-HPCC at SemEval-2024 Task10: Pre-trained Language Model for Emotion Discovery and Reasoning its Flip in Conversation

Chenyi Liang, Jin Wang, Xuejie ZhangNAACL 2024

TL;DR

This paper presents a fine-tuned DeBERTa model for the SemEval-2024 Task 10, which aims to identify emotions in conversations and detect emotional "flips." The authors introduce novel techniques like specialized [SEP] token usage and a KL divergence-based regularization (R-drop) to improve conversational emotion understanding and prevent overfitting.

What Problem Does This Paper Address?

Understanding and accurately interpreting human emotions in conversations is a significant challenge in Natural Language Processing (NLP). Conversations are dynamic, with emotions shifting rapidly due to context and subtle linguistic cues. The SemEval-2024 Task 10, "Emotion Discovery and Reasoning its Flip in Conversation" (EDiReF), specifically targets this by requiring models to:

  1. Classify emotional states for each utterance in a dialogue.
  2. Identify specific utterances that mark an emotional transition or "flip" within conversations.

This task is crucial for developing more sophisticated AI systems that can engage in more natural and empathetic human-computer interactions, such as advanced chatbots, virtual assistants, and mental health support tools.

Methods

The core of the proposed system is a DeBERTa (Decoding-enhanced BERT with disentangled attention) model, a powerful pre-trained transformer known for its effectiveness in understanding context and word positions. The authors enhance this base model with several key innovations:

  • Specialized [SEP] Token Usage: The model uses [SEP] tokens to clearly delineate individual utterances within a conversation, helping the model better grasp the flow and context of the dialogue.
  • Label Padding with -1: A specific padding strategy using -1 is employed to align utterances with their corresponding labels, ensuring that the model can handle variable-length conversational inputs effectively.
  • R-drop (Regularized Dropout): This technique, based on minimizing the KL divergence between model outputs under different dropout masks, is incorporated to prevent overfitting. R-drop encourages consistency in the model's predictions across different data sub-samples, leading to better generalization.
  • Focal Loss: To address class imbalance, a common issue in emotion detection where some emotions are much rarer than others, Focal Loss is used. This loss function dynamically adjusts the weight of each sample, focusing more on difficult-to-classify instances.
  • Back Translation and Synonym Replacement: For data augmentation, especially for the smaller subtask 1 dataset, these techniques were used to increase the diversity and size of the training data.

The architecture is adapted for the three subtasks: a multi-label sequence classification setup for Subtask 1, and binary sequence classification for Subtasks 2 and 3.

Datasets

The training and validation sets for the EDiReF task are derived from dialogues in various TV dramas.

  • Subtask 1: Contains 343 training sets with columns for episode, speakers, emotions, and utterances. It includes eight types of emotions.
  • Subtasks 2 & 3: Utilize larger datasets with 4893 and 4000 training sets, respectively, adding a "triggers" column to identify emotional transition points. Subtask 2 focuses on Hindi-English code-mixed dialogues, while Subtask 3 focuses on English conversations.

Key Insights and Ideas

  • DeBERTa's Suitability for Dialogue: The paper reinforces DeBERTa's strength in handling complex, contextual information, making it a strong candidate for dialogue-based NLP tasks.
  • The Power of Regularization and Loss Functions: The combination of R-drop and Focal Loss proved crucial for improving performance, especially in mitigating overfitting and handling imbalanced emotion labels. This highlights the importance of tailored training strategies beyond just the base model.
  • [SEP] Tokenization is Key for Dialogue Flow: The explicit use of [SEP] tokens to segment utterances significantly aids the model in understanding the sequential nature and context of conversations.
  • Data Augmentation's Impact: Techniques like back-translation and synonym replacement were vital for boosting performance on smaller datasets, demonstrating their effectiveness in enriching training data.

Key Results

The paper reports achieving specific results on the competition's official leaderboard, though exact final rankings are not detailed in the provided text.

  • Subtask 1: The deberta-base-focalloss-rdrop model achieved an F1 score of 0.39 on the validation set, outperforming baseline models and configurations using only focal loss or R-drop individually.
  • Subtask 2 & 3: The paper notes challenges with the debertav2-xlarge model, potentially due to its large parameter size and the small dataset, leading to a decline in performance. For Subtask 2, prediction issues arose, possibly from test set dialogues exceeding the 2048 token limit. For Subtask 3, the model performed well on identifying non-emotional triggers but was unsatisfactory for predicting emotional triggers.

Limitations and Future Work

  • Model Size vs. Dataset Size: The authors acknowledge that the debertav2-xlarge model's large parameter count might have hindered its performance on smaller datasets, making it difficult to learn effectively.
  • Truncation Length: For Subtask 2, dialogues exceeding the 2048 token truncation limit may have led to incomplete context capture and prediction shortfalls.
  • Language Specificity: The primary training of the base model in English might limit its effectiveness for code-mixed or non-English dialogues.
  • Future Directions: The authors express interest in exploring models like Longformer to better handle longer dialogues and potentially using multilingual models for improved performance on code-mixed tasks. They also aim to improve the model's ability to predict emotional triggers more accurately.

Summary updated

BibTeX
@inproceedings{liang2024ynu,
  author = {Chenyi Liang and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2024 Task10: Pre-trained Language Model for Emotion Discovery and Reasoning its Flip in Conversation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure