xiacui at SemEval-2025 Task 11: Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss

Xia CuiACL 2025

TL;DR

This paper tackles the challenge of imbalanced data in multi-label emotion detection using Transformer models. By introducing a simple weighted loss function, they show improved performance for models like BERT and BART, particularly for minority emotion classes, without the computational overhead of traditional resampling methods.

What Problem Does This Paper Address?

Detecting emotions in text is inherently complex due to the nuanced, context-dependent, and often ambiguous nature of human language. This task becomes even more challenging when dealing with short texts that can convey multiple emotions simultaneously, a scenario common in social media. The paper focuses on multi-label emotion detection, where a single piece of text can be associated with several emotion labels. A significant hurdle in this domain is data imbalance: some emotion classes are far more frequent in datasets than others. This imbalance can lead machine learning models to become biased towards the majority classes, performing poorly on less represented (minority) emotions. Traditional methods to combat this, like oversampling or undersampling, become complicated in multi-label settings, as adjusting one label's frequency can inadvertently affect others. This work aims to find a more effective and efficient way to handle this imbalance for Transformer-based models.

Methods

The core of the proposed solution is a weighted loss function applied to Transformer-based models. Instead of using the standard Binary Cross-Entropy (BCE) loss, which treats all classes equally, their approach dynamically adjusts the contribution of each class to the total loss. This is achieved by assigning higher weights to minority classes and lower weights to majority classes, effectively giving more "importance" to the underrepresented emotions during training. The weighting is based on the inverse frequency of each class in the training data.

The paper evaluates three popular Transformer architectures:

  • BERT (Bidirectional Encoder Representations from Transformers): Known for its deep bidirectional understanding of text.
  • RoBERTa (Robustly Optimized BERT Pretraining Approach): An optimized version of BERT with improved pretraining.
  • BART (Bidirectional and Auto-Regressive Transformers): A model that combines bidirectional and autoregressive pretraining, good for both understanding and generation.

These models are fine-tuned on the BRIGHTER dataset using this weighted loss function. The goal is to improve performance on minority classes without resorting to computationally expensive resampling techniques.

Datasets

The primary dataset used in this study is the BRIGHTER dataset (Muhammad et al., 2025a). Specifically, the authors focus on the English subset, which contains multi-label emotion annotations across 28 languages. The dataset includes annotations for five emotion classes: anger, fear, joy, sadness, and surprise. The training set was split into 70% for training and 30% for development (used as a test set due to inaccessible official test labels). No external datasets were used to augment the training data.

Key Insights and Ideas

  • Weighted loss is a viable alternative: The weighted loss function offers a computationally efficient way to address data imbalance in multi-label emotion detection, avoiding the complexities of resampling.
  • Transformer models benefit, but not uniformly: While BERT and BART show clear improvements with the weighted loss, RoBERTa's performance is less affected, suggesting its robust pretraining already mitigates some imbalance issues.
  • Minority class gains are nuanced: The weighted loss primarily boosts performance on high-frequency classes, with limited or no improvement on most minority emotion classes. This indicates that while the approach helps, it doesn't fully solve the problem for extremely underrepresented emotions.
  • Contextual ambiguity remains a challenge: The paper notes that "joy" and "surprise" often exhibit overlapping misclassifications, highlighting the difficulty in fine-grained emotion distinction in text, even for advanced Transformer models.

Key Results

The study reports that the weighted loss function generally improves performance across various metrics for BERT and BART. For instance, BART saw a significant increase in its Weighted F1 score from 0.6993 to 0.7350 and its Micro F1 score from 0.6961 to 0.7321 when using the weighted loss. BERT also showed improvements in Weighted F1 (0.7115 to 0.7228).

However, RoBERTa showed only marginal changes, with a slight decrease in Weighted F1 (0.7298 to 0.7270). This suggests RoBERTa's inherent robustness to data imbalance.

Interestingly, the improvements in Macro F1 and Sample F1 were primarily driven by gains in high-frequency classes, with limited impact on minority classes.

Limitations and Future Work

The authors acknowledge that the weighted loss function, while beneficial, disproportionately benefits high-frequency classes, potentially masking stagnation in minority emotion detection. They also note that RoBERTa's performance suggests it is less sensitive to data imbalance than BERT and BART, possibly due to its more robust pretraining.

Future work could involve:

  • Exploring the impact of this approach on different languages and datasets.
  • Investigating further methods to specifically boost minority emotion detection.
  • Addressing the contextual ambiguity between similar emotions like "joy" and "surprise."

Summary updated

BibTeX
@inproceedings{cui2025xiacui,
  author = {Xia Cui},
  title = {{xiacui at SemEval-2025 Task 11: Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure