SSN_Semeval10 at SemEval-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversations

Antony Rajesh A, Supriya Abirami A, Chandrabose Aravindan, Senthil Kumar BNAACL 2024

TL;DR

This paper tackles the challenge of recognizing emotions in Hindi-English code-mixed conversations using BERT-based transformer models. The authors fine-tuned pre-trained models like mBERT and indicBERT, achieving a competitive weighted F1-score of 0.4, ranking 16th out of 39 submissions for the SemEval-2024 Task 10. This work highlights the potential of transfer learning for nuanced emotion analysis in multilingual conversational contexts.

What Problem Does This Paper Address?

Understanding emotions in conversations is crucial for applications like sentiment analysis, mental health monitoring, and chatbot development. However, accurately identifying emotions becomes significantly more complex when conversations involve code-mixing, such as the common blend of Hindi and English used in many parts of India. This paper addresses the specific challenge of emotion recognition in such Hindi-English code-mixed dialogues, a task that requires models to grasp not only the semantic content but also the cultural nuances and linguistic interplay inherent in this communication style. The SemEval-2024 Task 10, specifically sub-task 1 (Emotion Recognition in Conversation - ERC), provides the benchmark for this investigation.

Methods

The core of the proposed approach lies in leveraging pre-trained transformer models, specifically BERT variants, for emotion classification. The authors employed two key models:

  • mBERT (Multilingual BERT): A well-established model pre-trained on a vast corpus of text in 104 languages, making it suitable for multilingual tasks.
  • indicBERT: A BERT model specifically designed and pre-trained for Indian languages, including Hindi.

These models were fine-tuned on the provided dataset for the SemEval task. The process involved:

  • Data Pre-processing: This included parsing JSON data, cleaning text by removing punctuation and stopwords, and crucially, incorporating the context of previous dialogues to inform the emotion prediction of the current utterance. Speaker names were also indexed for better context management.
  • Neural Architecture: The authors utilized a standard BERT architecture, incorporating self-attention mechanisms and feed-forward neural networks. They adopted an existing transformer model implementation for emotion recognition.
  • Training: Experiments were conducted on Google Colab using T4 GPUs. The models were trained with a batch size of 1 and a learning rate of 1e-6. The AdamW optimizer was used for weight updates, and cross-entropy loss was employed.
  • Evaluation: Performance was measured using precision, recall, and weighted F1-score on the validation set during training, and the final weighted F1-score was used for ranking.

Datasets

The paper utilizes the dataset provided for the SemEval-2024 Task 10 (EDiReF).

  • Training Set: Comprised 343 conversations with 8505 utterances.
  • Validation Set: Contained 46 conversations with 1354 utterances.

The dataset consists of conversations in Hindi-English code-mixed language, with each utterance labeled with one of the following emotions: 'neutral', 'contempt', 'sadness', 'fear', 'joy', 'surprise', 'anger', 'disgust'. The authors note that the emotion distribution in both datasets is skewed towards 'neutral'.

Key Insights and Ideas

  • Effectiveness of Transfer Learning: The study demonstrates that fine-tuning pre-trained BERT models (mBERT and indicBERT) is a viable and competitive strategy for emotion recognition in complex code-mixed conversational contexts.
  • Context is King: The importance of incorporating dialogue history (previous utterances and speaker information) is highlighted as a crucial step for better emotion understanding, moving beyond single-utterance analysis.
  • Challenges of Imbalanced Data: The significant skew towards 'neutral' emotions in the dataset is identified as a factor that might challenge models in accurately identifying less frequent emotions.
  • IndicBERT's Potential and Limitations: While indicBERT was designed for Indian languages, its performance was impacted by the dataset's use of transliterated Hindi versus the model's training on Hindi Unicode. This points to the critical need for dataset-model alignment in code-mixed scenarios.

Key Results

  • The system achieved a weighted F1-score of 0.4 on the validation set.
  • This score resulted in a 16th rank out of 39 submissions in SemEval-2024 Task 10, sub-task 1.
  • The top-ranked system achieved a weighted F1-score of 0.78, indicating a significant gap and room for improvement.
  • mBERT showed performance improvements up to around 7-8 epochs, while indicBERT's score remained more stable.
  • A confusion matrix analysis revealed that while mBERT performed reasonably well on 'neutral' emotions, it struggled with less frequent ones like 'disgust', 'contempt', 'fear', 'sadness', and 'surprise'.
  • The confusion matrix for indicBERT showed near-zero performance for all emotions except 'neutral', suggesting it did not effectively learn contextual representations from the provided dataset.

Limitations and Future Work

Limitations:

  • Nuanced Emotion Capture: The authors acknowledge that their system encountered challenges in accurately capturing nuanced emotional contexts, which impacted overall performance.
  • Data Imbalance: The skewed distribution of emotions in the dataset, with a heavy bias towards 'neutral', makes it difficult for models to learn and predict less frequent emotions.
  • IndicBERT's Training Data Mismatch: The use of transliterated Hindi in the dataset, while indicBERT was trained on Hindi Unicode, led to poor performance for indicBERT.

Future Work:

  • Dataset Expansion: Augmenting the dataset with more Hindi-English code-mixed data to expose models to a wider range of expressions.
  • Data Pre-processing Refinement: Translating Hindi-English utterances into plain English to potentially reduce ambiguity and improve model understanding.
  • Exploring Advanced Models: Investigating larger language models like LLAMA and GPT-2, as well as specialized models like HingBERT, for improved performance in Hindi-English code-mixed text analysis.

Summary updated

BibTeX
@inproceedings{a2024ssn,
  author = {Antony Rajesh A and Supriya Abirami A and Chandrabose Aravindan and Senthil Kumar B},
  title = {{SSN\_Semeval10 at SemEval-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure