MorphingMinds at SemEval-2024 Task 10: Emotion Recognition in Conversation in Hindi-English Code-Mixed Conversations

Monika VyasNAACL 2024

TL;DR

This paper tackles the challenge of recognizing emotions in Hindi-English code-mixed conversations, a complex linguistic phenomenon. The authors fine-tune transformer models and explore traditional machine learning approaches, ultimately finding that logistic regression, when combined with oversampling techniques, performs surprisingly well on this nuanced task.

What Problem Does This Paper Address?

Expressing emotions in multilingual settings, especially in code-mixed languages like Hindi-English, presents a significant hurdle for Natural Language Processing (NLP). Unlike monolingual text, code-mixed conversations blend linguistic structures, making it difficult for models to accurately capture emotional nuances. This paper specifically addresses the challenge of Emotion Recognition in Conversation (ERC) within this Hindi-English code-mixed context, aiming to improve NLP models' ability to understand and interpret these complex dialogues.

Methods

The research explores a multi-pronged approach to emotion recognition. Initially, traditional machine learning models like Support Vector Machines (SVM) and Logistic Regression (LR) were employed. The core of the methodology then shifted to leveraging advanced deep learning architectures, specifically fine-tuning a pre-trained XLM-ROBERTa model. A custom dataset was used for fine-tuning due to its unique label format. Feature extraction techniques like TF-IDF vectorization and N-gram analysis were utilized to capture linguistic patterns. To address class imbalance, oversampling techniques such as SMOTE (Synthetic Minority Over-sampling Technique) and ADASYN were implemented.

Datasets

The primary dataset used in this study is from the EDiReF shared task at SemEval 2024 (Kumar et al., 2024a). This dataset comprises dialogues sourced from a popular Indian comedy television series, featuring Hindi-English code-mixed conversations. The paper also mentions the creation of a custom dataset for fine-tuning the XLM-ROBERTa model due to label format incompatibilities.

Key Insights and Ideas

  • Logistic Regression's Resilience: Surprisingly, a traditional model like Logistic Regression, when combined with effective preprocessing and oversampling, outperformed more complex transformer-based models like XLM-Roberta on this specific code-mixed dataset. This highlights that for smaller, imbalanced datasets, simpler models can sometimes generalize better.
  • The Power of Oversampling: Techniques like SMOTE were crucial in mitigating class imbalance, a common issue in emotion recognition datasets. This directly led to improved model performance, demonstrating its importance for achieving robust results.
  • Code-Mixing Complexity: The paper underscores the inherent difficulty in processing code-mixed text, where linguistic structures from multiple languages intertwine, posing unique challenges for NLP models.
  • Preprocessing is Paramount: Meticulous data analysis and preprocessing, including handling missing values and ensuring accurate label mapping, were critical for successful model development and evaluation.

Key Results

The study reports that Logistic Regression achieved an accuracy of 33% and an F1-score of 35% after employing SMOTE oversampling and optimizing TF-IDF parameters. While these numbers might seem modest, they represent a significant improvement over initial attempts where models converged to predicting only the "neutral" label. The paper also notes that SVM achieved 30% accuracy and XLM-Roberta achieved 20% accuracy.

Limitations and Future Work

The authors acknowledge that the achieved accuracy (33%) and F1-score (35%) are still relatively low, indicating room for improvement. Future work will focus on integrating both word embeddings and TF-IDF features to create richer textual representations. The researchers also plan to explore Convolutional Neural Network (CNN) architectures for further refinement of the emotion detection process. Additionally, they aim to extend the methodology to accommodate multimodal data sources, such as text paired with audio or visual inputs, to broaden the application of emotion detection.

Summary updated

BibTeX
@inproceedings{vyas2024morphingminds,
  author = {Monika Vyas},
  title = {{MorphingMinds at SemEval-2024 Task 10: Emotion Recognition in Conversation in Hindi-English Code-Mixed Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure