Innovators at SemEval-2024 Task 10: Revolutionizing Emotion Recognition and Flip Analysis in Code-Mixed Texts

Abhay Shanbhag, Suramya Jadhav, Shashank Rathi, Siddhesh Pande, Dipali KadamNAACL 2024

TL;DR

This paper tackles the challenge of emotion recognition and "emotion flip" detection in code-mixed languages, specifically Hinglish (Hindi-English). The authors developed innovative transformer-based and ensemble methods, achieving impressive results, including a 2nd rank in Track 2 of the SemEval-2024 Task 10. Their work highlights the critical role of translation and contextual understanding in accurately analyzing emotions in these dynamic linguistic contexts.

What Problem Does This Paper Address?

The increasing prevalence of code-mixed languages, particularly on social media platforms in India, presents a significant challenge for natural language processing (NLP) tasks like emotion recognition. Hinglish, a blend of Hindi and English, often leads to misidentification of parts of speech and requires a deeper understanding of context than traditional NLP models designed for monolingual text. Specifically, this paper addresses two key problems within conversational analysis:

  1. Emotion Recognition in Conversation (ERC): Assigning the correct emotion to each utterance in a Hinglish conversation from a predefined set of emotions.
  2. Emotion Flip Reasoning (EFR): Identifying the specific utterance(s) that trigger a change in emotion within a multi-party conversation in Hinglish and English.

The scarcity of labeled data and the inherent complexity of code-mixed languages make these tasks particularly difficult for existing state-of-the-art NLP models.

Methods

The authors employed a multi-pronged approach, leveraging both traditional machine learning and advanced transformer-based models.

For Track 1 (Emotion Recognition), they explored:

  • Ensemble Techniques: Utilizing classic machine learning models such as Decision Trees, SVM, Logistic Regression, and Multinomial Naive Bayes, complemented by hyperparameter tuning.
  • Transformer-based Models: Employing advanced models like XLM-Roberta (XLMR), DistilRoberta, and T5 from Hugging Face's transformer library. A key innovation here was a two-stage pipeline: first, translating Hinglish to English, and then performing emotion detection on the translated English text. This pipeline was fine-tuned on the Hinglish TOP dataset.

For Tracks 2 and 3 (Emotion Flip Reasoning), they developed two novel algorithms:

  • Algorithm 1: Designed to identify the last emotion flip of every user by analyzing entire conversation episodes and outputting trigger presence.
  • Algorithm 2: Specifically focused on identifying trigger points in conversations, marking the second-to-last conversation as a trigger.

Additionally, to address class imbalance in the datasets, they utilized oversampling and undersampling techniques, finding oversampling to be more effective.

Datasets

  • SemEval-2024 Task 10 Dataset: This dataset was provided by the organizers for all three tracks.
  • Track 1: Contained conversational episodes with speaker-utterance pairs and target emotions.
  • Tracks 2 & 3: Included utterances, emotions, and a "trigger" label (1 for the last emotion flip, 0 otherwise). Track 3 used English utterances, while Track 2 used Hinglish.
  • Hinglish-TOP Dataset: Used for fine-tuning the Hinglish-to-English translation component of their pipeline.

Key Insights and Ideas

  • Translation as a Bridge: Translating Hinglish to English before emotion analysis significantly improves performance. This suggests that while code-mixing is common in speech, standard NLP models perform better on more structured, monolingual text.
  • Context is King: The paper strongly emphasizes that for code-mixed languages, context is crucial for accurate emotion detection, unlike in pure languages where a single sentence might suffice.
  • Oversampling for Imbalance: Minority oversampling proved to be a highly effective strategy for mitigating class imbalance, leading to better representation and recognition of underrepresented emotions.
  • Transformer Power: Advanced transformer models like XLM-Roberta and T5, when fine-tuned on relevant datasets, demonstrate superior capabilities in handling the nuances of code-mixed text.

Key Results

The authors achieved significant success in the SemEval-2024 Task 10:

  • Track 2 (Emotion Flip Reasoning): Secured 2nd rank with an F1 score of 0.79.
  • Track 3 (Emotion Flip Reasoning): Achieved 7th rank with an F1 score of 0.68.
  • Track 1 (Emotion Recognition): Obtained an F1 score of 0.28.

Their ensemble methods for Track 1 showed varying F1 scores, with SVM (Oversampled) reaching 0.2830 and Logistic Regression - Softmax (Oversampled) also achieving 0.2809. The XLMR approach yielded an F1 score of 0.2626, and their pipeline approach achieved 0.2688.

Limitations and Future Work

  • Data Imbalance: While oversampling helped, the inherent class imbalance in the dataset (e.g., a 30:1 ratio between dominant and minority classes) remains a significant challenge.
  • False Positives/Negatives: Algorithm 1 for Tracks 2 and 3 showed a considerable number of false positives (959) compared to Algorithm 2 (68), although false negatives were comparable.
  • Future Work: The paper implicitly suggests that further research into more robust translation models for code-mixed languages and exploring more sophisticated contextual understanding mechanisms could lead to even better performance. The success of the transformer pipeline also points towards continued exploration of these architectures for code-mixed NLP tasks.

Summary updated

BibTeX
@inproceedings{shanbhag2024innovators,
  author = {Abhay Shanbhag and Suramya Jadhav and Shashank Rathi and Siddhesh Pande and Dipali Kadam},
  title = {{Innovators at SemEval-2024 Task 10: Revolutionizing Emotion Recognition and Flip Analysis in Code-Mixed Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure