TL;DR
This paper explores state-of-the-art encoder models for multi-label emotion detection across 28 languages. They found that using pre-trained embeddings with tree-based classifiers (like BGE + CatBoost) significantly outperforms traditional end-to-end fine-tuning of transformer models, especially when prompts are tailored to individual emotions.
What Problem Does This Paper Address?
Detecting emotions in text is inherently challenging due to the subjective and context-dependent nature of emotional expression. This task becomes even more complex when dealing with multi-label emotion detection (identifying multiple emotions simultaneously) across a multilingual dataset, especially one with many low-resource languages. Existing approaches often struggle with class imbalance and the nuances of diverse linguistic and cultural contexts. The authors aim to find the most effective way to leverage advanced encoder models for this complex task.
Methods
The researchers investigated two primary strategies:
- Classifier-Only Training: This involves using pre-trained encoder models (like Jina, BGE, mE5, and XLM-R) to generate text embeddings, and then training a separate machine learning classifier (SVC, Logistic Regression, CatBoost, XGBoost) on these embeddings. The encoders are "frozen" during this process.
- End-to-End Fine-Tuning: This approach involves fine-tuning the entire transformer model for the emotion detection task. They explored two sub-strategies:
- Multiple Head Approach: A single output layer predicts all emotion categories simultaneously.
- [SEP] Token Separation: The input is formatted as
<sentence> [SEP] <emotion>, treating each emotion as a binary classification problem.
To address class imbalance, they experimented with Focal Loss and Asymmetric Loss functions. They also explored different prompting strategies for the encoder models, ranging from general prompts to emotion-specific ones.
Datasets
The primary dataset used is the BRIGHTER dataset (Muhammad et al., 2025a), which is specifically designed for multilingual multi-label emotion classification and includes a significant number of low-resource languages.
Key Insights and Ideas
- Embeddings Outperform Fine-Tuning: The most striking finding is that using pre-trained embeddings from models like BGE and mE5, combined with a simple classifier like CatBoost, significantly outperforms fine-tuning larger transformer models like mBERT and XLMR. This suggests that the quality of the embeddings is paramount for this task.
- Prompt Engineering Matters: Tailoring prompts to be emotion-specific (e.g., asking the model to identify a single emotion at a time) led to noticeable improvements, even with simpler classifiers. This highlights the importance of guiding the encoder's output.
- Multilingual vs. Monolingual: Training on all languages (ALL) versus training language-specifically (LANG) did not yield a significant difference in performance. This implies that for this specific task and dataset, the benefits of multilingual training might be marginal compared to language-specific approaches, which are also more computationally efficient.
- Loss Functions are Similar: Both Focal Loss and Asymmetric Loss performed similarly in mitigating class imbalance, suggesting that other factors (like the encoder and classifier choice) had a greater impact.
Key Results
- The best-performing model was an ensemble of BGE models with CatBoost as the classifier, achieving an average F1-macro score of 56.58 across all languages on the final leaderboard.
- In quantitative comparisons, prompt-based encoder models (BGE and mE5) with classifiers outperformed fully fine-tuned models, with BGE achieving an average F1-macro score of 47.3% compared to 37.7% for fully fine-tuned models.
- The paper presents detailed performance tables (Tables 4-13) showing F1-macro scores for various model configurations across different languages, demonstrating the superiority of BGE and mE5-based approaches.
Limitations and Future Work
- Limited Qualitative Analysis: The authors acknowledge a lack of extensive qualitative analysis due to their limited proficiency in many of the dataset's languages. This means their conclusions are primarily based on quantitative metrics.
- Future Work: The success of prompt engineering suggests that further research into optimizing prompts could yield even better results. The paper also implicitly points towards exploring more advanced ensemble techniques and potentially different classifier architectures.
Summary updated