Tue-JMS at SemEval-2025 Task 11: KReLax: An Ensemble-Based Approach for Multilingual Emotion Detection and Addressing Data Imbalance

Jingyu Han, Megan Horikawa, Suvi LehtosaloACL 2025

TL;DR

This paper introduces KReLax, a multilingual ensemble model designed to tackle the challenge of emotion detection across diverse languages, especially those with limited resources. By combining multiple BERT-based encoders and employing data augmentation, KReLaX achieves strong performance in multi-label emotion classification and intensity prediction, while also highlighting the difficulties of cross-lingual generalization.

What Problem Does This Paper Address?

Emotion detection, the task of identifying the emotional tone or state conveyed in text, has seen significant progress, but primarily in high-resource languages like English. This leaves a substantial gap for low-resource languages, where data and robust models are scarce. SemEval-2025 Task 11 specifically aims to bridge this gap by focusing on perceived emotions in underrepresented languages across three tracks: multi-label emotion detection, emotion intensity classification, and cross-lingual emotion detection. The core challenge lies in building models that can generalize effectively across languages with varying linguistic structures and limited annotated data, while also handling the inherent imbalance of emotion occurrences within datasets.

Methods

The KReLax system is built upon a robust ensemble learning approach. It leverages three pre-trained multilingual BERT variants: XLM-RoBERTa, LaBSE, and RemBERT. These models are fine-tuned on the available training data for each language. A key innovation is the use of a weighted soft voting layer for the final classification. This layer assigns weights to the predictions of individual models based on their performance (specifically, the squared average F1 score), allowing better-performing models to have a greater influence on the final decision.

To combat data imbalance, the authors employ class weighting, calculating weights based on the inverse class frequency. For data augmentation, they utilize a multilingual LLM to generate rephrased versions of the training texts, focusing on synonym replacement and random swap techniques. This augmentation is applied judiciously, with augmented data comprising less than 20% of the training set to avoid overwhelming the original data's characteristics. For emotion intensity classification (Track B), data augmentation was intentionally avoided to prevent unintended alterations to emotional nuances.

Datasets

The primary datasets used in this work are part of the BRIGHTER collection, created for SemEval-2025 Task 11. This collection encompasses data for 28 languages, with a significant focus on low-resource languages. Additionally, the EthioEmo dataset was used to supplement the data for four languages. The task is divided into three tracks:

  • Track A (Multi-Label Emotion Detection): Predicts the presence or absence of emotions (anger, disgust, fear, joy, sadness, surprise) per text.
  • Track B (Emotion Intensity Classification): Predicts the intensity of each emotion on a scale of 0 to 3.
  • Track C (Cross-Lingual Emotion Detection): Tests the model's ability to generalize to unseen languages without specific training data for those languages.

Key Insights and Ideas

  • Ensemble Power: Combining multiple BERT-based models with a weighted voting mechanism significantly boosts performance, especially in multi-label classification. This suggests that diverse model perspectives can capture different facets of emotional expression.
  • Data Augmentation's Role: Leveraging LLMs for data augmentation proved effective in improving robustness for multi-label classification, demonstrating the potential of generative models in low-resource NLP tasks.
  • Intensity vs. Labeling: The authors found that data augmentation techniques suitable for multi-label classification were not directly applicable to emotion intensity prediction, as they could inadvertently alter the subtle nuances of emotional intensity. This highlights a distinct challenge in modeling intensity.
  • Cross-Lingual Hurdles: Zero-shot cross-lingual performance remains a significant challenge. The results suggest that linguistic similarity between training and unseen languages plays a crucial role in generalization, underscoring the need for more advanced cross-lingual transfer learning techniques.
  • Emotion Imbalance: Certain emotions like Fear, Disgust, and Surprise were consistently underrepresented and harder to detect, likely due to their reliance on subtle cues and context that vary across cultures and languages.

Key Results

  • Track A (Multi-Label Emotion Detection): KReLax achieved strong results, ranking 2nd in Tatar and showing competitive performance in Hindi, Russian, Marathi, and Spanish. The system achieved an average F1 score of .636 across 24 languages.
  • Track B (Emotion Intensity Classification): The system performed well, securing 4th place in Hausa and 5th place in Amharic, with an overall average Pearson correlation of .6724.
  • Track C (Cross-Lingual Emotion Detection): This track proved challenging. The best performance was in Indonesian, with a macro F1 score of .5077, ranking 7th.

Limitations and Future Work

The authors acknowledge that their system struggled with zero-shot performance in Track C, indicating a need for improved cross-lingual generalization. They also note that data augmentation for Track B was not feasible due to the risk of altering emotional intensity.

Future work could explore:

  • Developing more sophisticated data augmentation strategies specifically for emotion intensity.
  • Investigating decoder-based architectures for potentially better performance.
  • Further research into techniques that enhance cross-lingual generalization, especially for languages with significant typological differences from the training data.
  • Addressing label misalignment issues in datasets.

Summary updated

BibTeX
@inproceedings{han2025tue,
  author = {Jingyu Han and Megan Horikawa and Suvi Lehtosalo},
  title = {{Tue-JMS at SemEval-2025 Task 11: KReLax: An Ensemble-Based Approach for Multilingual Emotion Detection and Addressing Data Imbalance}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure