Zero_Shot at SemEval-2025 Task 11: Fine-Tuning Deep Learning and Transformer-based Models for Emotion Detection in Multi-label Classification, Intensity Estimation, and Cross-lingual Adaptation

Ashraful Islam Paran, Sabik Aftahee, Md. Refaj Hossan, Jawad Hossain, Mohammed Moshiul HoqueACL 2025

TL;DR

This paper tackles the challenge of emotion detection in low-resource languages (LRLs) by fine-tuning deep learning and transformer models. They achieved state-of-the-art results on the SemEval-2025 Task 11 across multi-label classification, intensity estimation, and cross-lingual adaptation, demonstrating the power of models like XLM-R and BiLSTM+CNN for nuanced text understanding.

What Problem Does This Paper Address?

Recognizing emotions in text is crucial for applications ranging from mental health monitoring to virtual assistants. However, most existing research and datasets focus on high-resource languages, leaving a significant gap for low-resource languages (LRLs). This disparity hinders the development of universally applicable emotion detection systems. The SemEval-2025 Shared Task 11 aimed to bridge this gap by focusing on approximately 32 LRLs and presenting three tracks: multi-label emotion classification, emotion intensity estimation, and cross-lingual emotion detection. This paper addresses this challenge by exploring various machine learning, deep learning, and transformer-based models to achieve robust emotion detection in these underrepresented languages.

Methods

The authors explored a range of models, categorized into three main groups:

  • Machine Learning (ML) Models: Logistic Regression (LR), Support Vector Machines (SVM), Naive Bayes (NB), and Random Forest (RF) were used. Hyperparameter tuning was performed using GridSearchCV.
  • Deep Learning (DL) Models: Convolutional Neural Networks (CNN), BiLSTM+CNN, and BiLSTM+BiGRU were employed. These models leverage sequential dependencies in text through recurrent and convolutional layers.
  • Transformer-based Models: XLM-R, mBERT, and ModernBERT were fine-tuned. These models are known for their powerful attention mechanisms and multilingual capabilities, making them suitable for cross-lingual tasks.

For feature extraction, TF-IDF was used for ML models, while GloVe and Keras embeddings were used for DL models. Data preprocessing involved cleaning text, removing emojis and special characters, lowercasing, tokenization, and stopword removal.

Datasets

The study utilized datasets for the SemEval-2025 Shared Task 11, which included approximately 32 low-resource languages. The specific languages mentioned for evaluation include Amharic, Hindi, Igbo, Marathi, Russian, Algerian Arabic, Chinese, Hausa, Oromo, and Somali. The paper notes that Track C (cross-lingual emotion detection) specifically lacked a labeled training corpus, relying on transfer learning. Appendix A provides detailed class-wise distributions for the train, validation, and test sets across different languages for Tracks A and B.

Key Insights and Ideas

  • Transformer Dominance: Transformer-based models, particularly XLM-R, significantly outperformed both traditional ML and DL models across most tasks and languages, especially for multi-label classification and intensity estimation. This highlights their superior ability to capture nuanced linguistic and contextual information.
  • BiLSTM+CNN for Cross-lingual: While transformers excelled in other areas, the BiLSTM+CNN model showed strong performance in the cross-lingual emotion detection task (Track C), suggesting its effectiveness in transferring learned representations across languages.
  • LRL Challenges: The study underscores the inherent difficulties in LRLs, including class imbalance (e.g., fear being underrepresented) and the subjective nature of emotion intensity labeling, leading to inconsistencies.
  • Cultural Nuances Matter: The analysis reveals that cultural differences and linguistic structures significantly impact emotion expression and detection, especially in LRLs. Models struggle with implicitly expressed emotions or those heavily coded by culture.
  • Bias Towards Negativity: A notable observation is the model's bias towards predicting negative instances (class 0), potentially due to data skewness or calibration issues, as seen in the confusion matrices.

Key Results

  • Track A (Multi-label Emotion Classification): XLM-R achieved the highest F1 scores in four out of five languages, including Hindi (84.60), Marathi (78.74), and Russian (84.38). m-BERT performed best for Igbo (49.35).
  • Track B (Emotion Intensity Prediction): XLM-R also delivered the best results for emotion intensity detection across most languages.
  • Track C (Cross-lingual Emotion Detection): BiLSTM+CNN was found to be superior for this task, outperforming other architectures in most cases.
  • ML vs. DL vs. Transformers: Across the board, transformer models consistently surpassed ML and DL models, with XLM-R being a standout performer. SVM showed competitive results among ML models, and CNN+BiLSTM was a strong contender among DL models.

Limitations and Future Work

  • Subjective Labeling: The emotion intensity prediction task faces challenges due to subjective labeling, leading to dataset inconsistencies.
  • Multilingual Nuances: Capturing fine-grained emotion variations and overlapping emotions in multilingual and code-mixed scenarios remains difficult due to cultural and linguistic differences.
  • Lack of Labeled Training Data (Track C): The absence of a labeled training dataset for cross-lingual tasks limits generalization and heavily relies on transfer learning.
  • Class Imbalance: Underrepresentation of certain emotions affects model learning and prediction accuracy for rare emotions.
  • Future Work: The authors plan to explore enhancing model generalizability by merging self-supervised learning and contrastive learning. They also aim to research domain adaptation methods and data augmentation strategies to improve emotion recognition in LRLs and multilingual social media settings.

Summary updated

BibTeX
@inproceedings{paran2025zero,
  author = {Ashraful Islam Paran and Sabik Aftahee and Md. Refaj Hossan and Jawad Hossain and Mohammed Moshiul Hoque},
  title = {{Zero\_Shot at SemEval-2025 Task 11: Fine-Tuning Deep Learning and Transformer-based Models for Emotion Detection in Multi-label Classification, Intensity Estimation, and Cross-lingual Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure