TL;DR
This paper presents the ISDS-NLP team's approach to SemEval-2024 Task 10, focusing on emotion recognition and emotion-flip reasoning in mixed-language conversations. They achieved a respectable 12th place using fine-tuned Transformer models, particularly excelling with Masked Language Models (MLMs) over large generative models for this specific task.
What Problem Does This Paper Address?
Identifying emotions in conversations is crucial for understanding human interaction, especially in contexts like customer service or virtual assistants where managing conflict is key. SemEval-2024 Task 10 specifically targets this by asking participants to not only label the emotion of each utterance but also to identify the "trigger" utterance that causes an "emotion flip" (e.g., a shift from neutral to anger). This is particularly challenging in code-mixed (Hindi-English) conversations, where linguistic nuances can be complex. The task aims to build more empathetic and responsive AI systems.
Methods
The ISDS-NLP team explored two main Transformer-based approaches for Subtask 1 (emotion recognition):
- Masked Language Modeling (MLM): They utilized pre-trained multilingual BERT-like models (specifically
xlm-roberta-largeandmdeberta-v3) and fine-tuned them. This involved: - Input Handling: Analyzing utterance lengths and finding that a maximum sequence length of 55 tokens yielded the best results. They found that preprocessing like lemmatization or stopword removal was detrimental, suggesting these elements carry useful information.
- Classifier Selection: Experimenting with various classifiers, with fully connected layers proving most effective due to their ability to model complex relationships.
- Fine-tuning Strategy: Employing a two-stage fine-tuning process: first training only the classifier with a higher learning rate, then fine-tuning both the classifier and the transformer's final layer with a lower learning rate to prevent overfitting.
- Causal Language Modeling (CLM): They experimented with the state-of-the-art Mistral 7B Instruct V0.2 model using zero-shot and few-shot prompting techniques.
Datasets
The datasets used were part of the SemEval-2024 Task 10. They consisted of English and Hindi code-mixed conversations for Subtasks 1 and 2, and English-only conversations for Subtask 3. The dataset is described as relatively small, particularly for the training set of Subtask 1. The authors note that the dataset is based on MELD, an existing emotion recognition dataset, augmented with triggers for the emotion-flip task.
Key Insights and Ideas
- MLMs Outperform Generative Models for Sentence Classification: Despite the rise of large generative models like Mistral, fine-tuned MLMs (like
xlm-roberta-large) demonstrated superior performance for sentence-level emotion classification in this task. - Data Size Matters: The authors suggest that with more data, their MLM models could achieve significantly higher scores, hinting at the dataset's limitations.
- Punctuation and Stopwords are Informative: Unlike typical NLP preprocessing, removing punctuation and stopwords negatively impacted their model's performance, indicating these elements are crucial for emotion recognition.
- Overfitting is a Major Concern: The model's performance degraded with excessive fine-tuning epochs, highlighting the importance of careful hyperparameter tuning and strategies to combat overfitting.
- Class Imbalance is a Challenge: The model overpredicts the "neutral" emotion due to its prevalence in the training data, leading to misclassifications of other emotions.
Key Results
- Subtask 1 Performance: The ISDS-NLP team achieved a weighted F1 score of 0.43 on Subtask 1, placing them 12th on the leaderboard.
- Model Comparison: The fine-tuned
xlm-roberta-largemodel achieved the best test score of 0.43, outperforming other MLM models and the zero-shot/few-shot Mistral models. - Mistral Performance: Mistral's zero-shot performance was 0.32, and few-shot was 0.31, indicating it struggled in this specific classification task compared to fine-tuned MLMs.
- Emotion Distribution: The most predominant emotion in the dataset is "neutral," followed by "joy" and "anger." The model performed best on "neutral," "joy," and "sadness," but struggled with under-represented emotions like "disgust."
Limitations and Future Work
- Class Imbalance: The authors acknowledge the model's tendency to overpredict the "neutral" emotion due to class imbalance. They suggest future work should explore more robust techniques like oversampling or undersampling.
- Emotion-Flip Tasks: While achieving good results on Subtask 1, their performance on the emotion-flip detection tasks (Subtasks 2 and 3) was less strong, placing them 14th on Subtask 2.
- Hybrid Architectures: They plan to investigate hybrid Transformer-LSTM architectures for a more nuanced understanding of emotion-flip triggers.
- Broader Context: Incorporating broader conversational context through multi-turn analysis is suggested to enhance model capabilities.
- Newer Generative Models: The authors note that newer causal models like Mixtral and Solar might perform better on this type of task.
Summary updated