Zero at SemEval-2025 Task 11: Multilingual Emotion Classification with BERT Variants: A Comparative Study

Revanth Gundam, Abhinav Marri, Radhika MamidiACL 2025

TL;DR

This paper tackles the challenging task of multilingual emotion detection and intensity prediction using BERT-based models. By comparing various fine-tuning strategies and model sizes on the BRIGHTER dataset, the authors demonstrate that larger, fine-tuned transformer models significantly outperform static embeddings, offering a robust approach to understanding emotions across diverse languages.

What Problem Does This Paper Address?

Understanding human emotions expressed in text is crucial for many NLP applications, from sentiment analysis to building more empathetic chatbots. However, accurately detecting emotions in text is complex due to the nuances of language, cultural differences, and the fact that multiple emotions can be present simultaneously (multi-label classification). Furthermore, quantifying the intensity of these emotions adds another layer of difficulty. This paper addresses these challenges by focusing on two specific tasks: multi-label emotion classification (identifying which of six emotions are present) and emotion intensity prediction (ranking the strength of a given emotion) across multiple languages. A significant hurdle in this domain is class imbalance, where some emotions are far more common than others, making it difficult for models to learn effectively.

Methods

The authors explore a range of methods, starting with baseline approaches and progressing to state-of-the-art transformer models.

  • Baseline: They begin by using static word embeddings like GloVe, combined with logistic regression classifiers. This serves as a foundational comparison point.
  • Transformer Embeddings (Frozen): Next, they leverage pre-trained transformer models like BERT (bert-base-uncased) and RoBERTa, using their contextualized embeddings without further training. This captures more semantic meaning than static embeddings.
  • Fine-tuning Transformers: The core of their approach involves fine-tuning transformer models. This allows the models to adapt their learned representations specifically to the emotion detection and intensity prediction tasks on their chosen dataset. They experiment with different sizes of BERT:
  • bert-base-uncased: A standard BERT model.
  • RoBERTa: A variant of BERT known for improved performance.
  • bert-large-uncased: A larger version of BERT with more parameters, hypothesized to capture more complex patterns.
  • Handling Class Imbalance: To combat class imbalance, they adopt a strategy of splitting the multi-label classification problem into independent binary classification tasks for each emotion. This means training a separate classifier for "joy," another for "sadness," and so on, rather than a single multi-label classifier.
  • Multilingual Adaptation: For multilingual classification, they utilize Multilingual BERT (mBERT), a model pre-trained on a vast corpus of text in over 100 languages, enabling cross-lingual transfer learning.

Datasets

The primary dataset used in this study is BRIGHTER.

  • BRIGHTER: This is a multilingual, emotion-annotated dataset spanning 28 languages, including low-resource languages from Africa and Asia, as well as high-resource languages like English. It provides text snippets labeled with the presence (or absence) of six emotions: joy, sadness, fear, anger, surprise, and disgust. For the intensity prediction task, labels range from 0 to 3. The paper focuses on English, Spanish, Russian, and Romanian subsets for their experiments.

Key Insights and Ideas

  • Fine-tuning is King: The study strongly reinforces that fine-tuning pre-trained transformer models is significantly more effective for emotion classification than using static embeddings or even frozen transformer embeddings. This highlights the importance of task-specific adaptation.
  • Bigger is Often Better: Larger transformer models, specifically bert-large, consistently achieve the best performance, suggesting that increased model capacity and deeper contextual understanding are crucial for nuanced emotion detection.
  • Decomposition for Imbalance: Splitting the multi-label task into individual binary classifiers for each emotion is an effective strategy to mitigate class imbalance and improve performance.
  • Multilingual Transfer Works, But Varies: mBERT enables cross-lingual generalization, but its effectiveness is influenced by the representation of specific languages in its pre-training data, leading to performance variations across languages.

Key Results

  • Binary Classification: Fine-tuned bert-large and RoBERTa models achieved substantially higher F1 scores than GloVe embeddings and frozen BERT models. bert-large attained the highest average F1 scores across all emotions.
  • Emotion Intensity Prediction: On the English dataset, the fine-tuned bert-large model achieved the highest average Pearson correlation coefficient (r = 0.6129) for predicting emotion intensity, outperforming frozen BERT embeddings (r = 0.5315).
  • Multilingual Performance: In multilingual classification, Spanish showed the highest overall performance in terms of macro F1 scores, suggesting that language representation in pre-training data plays a critical role. However, performance varied significantly across languages, with lower F1 scores for surprise and anger in some cases.

Limitations and Future Work

  • Class Imbalance: The paper acknowledges that class imbalance remains a significant challenge, particularly for underrepresented emotions. This leads to biased learning.
  • Future Work: The authors suggest exploring techniques like oversampling, under-sampling, and synthetic data generation to create more balanced datasets and improve model robustness.
  • Model Architecture: The study primarily focuses on BERT-based models.
  • Future Work: They propose investigating more advanced architectures like DeBERTa, T5, or GPT-based models, which might offer better performance due to disentangled attention or generative learning strategies.
  • Supervised Learning Dependency: The current approach relies on supervised learning, which requires labeled data.
  • Future Work: For low-resource settings, they suggest exploring semi-supervised and self-supervised learning techniques to leverage unlabeled data. Pre-training on larger, diverse emotion-rich corpora before fine-tuning could also enhance adaptability.

Summary updated

BibTeX
@inproceedings{gundam2025zero,
  author = {Revanth Gundam and Abhinav Marri and Radhika Mamidi},
  title = {{Zero at SemEval-2025 Task 11: Multilingual Emotion Classification with BERT Variants: A Comparative Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure