TL;DR
This paper details UMUTeam's approach to SemEval-2024 Task 10, focusing on detecting and reasoning about emotional shifts in conversations, particularly in code-mixed Hindi-English dialogues. They achieved competitive results by fine-tuning pre-trained Transformer models, demonstrating the effectiveness of BERT for this complex task.
What Problem Does This Paper Address?
Understanding and responding to human emotions is crucial for building more empathetic and effective conversational AI systems. While emotion recognition in text has been studied, analyzing emotional changes within a conversation, especially in multilingual or code-mixed settings, presents a significant challenge. This task, EDiReF (Emotion Discovery and Reasoning its Flip in Conversation), aims to address this by:
- Emotion Recognition in Conversation (ERC): Identifying emotions in utterances within a dialogue.
- Emotion Flip Reasoning (EFR): Identifying the specific utterances that trigger a change in a speaker's emotional state.
The paper highlights the under-representation of code-mixed conversations in previous ERC research and the need for systems that can handle the nuances of language switching and emotional dynamics in such dialogues.
Methods
The UMUTeam's approach is centered around fine-tuning pre-trained Transformer-based models. The core methodology involves:
- Data Preprocessing:
- For Subtask 1 (Hindi-English code-mixed ERC), utterances were translated into English to leverage English-centric language models.
- Utterances were grouped by speaker to provide conversational context.
- For each target utterance, previous and subsequent utterances from the same speaker were concatenated with a semicolon (e.g.,
U2; U3; U4). - For Subtasks 2 and 3 (EFR), the emotion of each concatenated utterance was also included (e.g.,
U2-e2; U3-e3; U4-e4). - Model Fine-tuning:
- Pre-trained models (XLM-ROBERTa-base, DeBERTa-V3-base, and BERT) were used as the base.
- A sequence classification layer was added on top of these models.
- The models were fine-tuned using a weighted F1 score as the evaluation metric, accounting for class imbalance.
- Hyperparameters: Consistent hyperparameters were used across all subtasks: batch size of 8, 10 epochs, learning rate of 2e-5, and weight decay of 0.01.
Datasets
The authors utilized the datasets provided by the organizers for the SemEval-2024 Task 10 (EDiReF). These datasets cover three subtasks:
- Subtask 1: ERC in Hindi-English code-mixed conversations.
- Subtask 2: EFR in Hindi-English code-mixed conversations.
- Subtask 3: EFR in English conversations.
The paper notes that the training set for Subtask 1 is unbalanced, with the "neutral" emotion occupying the highest percentage.
Key Insights and Ideas
- Transformer Powerhouse: Fine-tuning pre-trained Transformer models like BERT, XLM-R, and DeBERTa proved effective for both emotion recognition and emotion flip reasoning, even in challenging code-mixed scenarios.
- Context is King (for the Speaker): Grouping utterances by speaker and concatenating adjacent turns from that speaker was a key strategy to provide relevant context for emotion analysis, rather than incorporating context from all interlocutors. This suggests that understanding an individual's emotional trajectory is paramount.
- Code-Mixing is a Hurdle, but Solvable: The success in Subtask 1, which specifically targets Hindi-English code-mixed data, demonstrates that existing NLP models can be adapted to handle linguistic complexity, though translation to a common language (English) was a helpful preprocessing step.
- EFR is Tricky: Identifying the cause of an emotion flip (EFR) is more challenging than simply recognizing the emotion itself. The results for EFR subtasks were generally lower than for ERC.
Key Results
The UMUTeam achieved the following results using their fine-tuned BERT model:
- Subtask 1 (ERC, Hindi-English code-mixed): 13th best result with a weighted F1 score of 43%. (XLM-R performed slightly better with 42.878% vs BERT's 42.691% in the paper's Table 2, but the abstract states 43% for BERT).
- Subtask 2 (EFR, Hindi-English code-mixed): 6th best result with an F1 score of 26%.
- Subtask 3 (EFR, English): 15th best result with an F1 score of 22%.
BERT was chosen as the final model because it outperformed others in all three subtasks, with only a minor difference in Subtask 1 compared to XLM-ROBERTa.
Limitations and Future Work
- Class Imbalance: The "neutral" emotion in Subtask 1's training data was highly dominant, leading to potential confusion and misclassification. The disgust emotion was not correctly identified in any case in the confusion matrix.
- EFR Hyperparameter Sensitivity: The paper suggests that the XLM-R and DeBERTa models struggled with EFR possibly due to the same hyperparameters being used for all subtasks. Future work could involve exploring different hyperparameter tuning strategies, such as a smaller learning rate, for EFR tasks.
- Future Application: The authors plan to apply their emotion flip inference techniques to political communication datasets to gain deeper insights into the ideologies and motivations of political figures.
Summary updated