TL;DR
This paper presents VerbaNexAI's approach to SemEval-2025 Task 11, focusing on multi-label emotion classification in English text. They leverage a RoBERTa model, enhanced with data augmentation and a class imbalance mitigation strategy, achieving competitive results in a challenging task.
What Problem Does This Paper Address?
The paper tackles the complex problem of automatically detecting and classifying emotions in text, a crucial area for understanding human interaction in the digital age. Specifically, it addresses SemEval-2025 Task 11, which focuses on multi-label classification of five basic emotions (joy, sadness, fear, anger, surprise) in English text. The core challenge lies in the inherent complexity of emotional language, especially when emotions are implicit, expressed through idiomatic language, or when datasets suffer from class imbalance, making it difficult for models to accurately identify less frequent emotions.
Methods
The VerbaNexAI system is built upon the pre-trained RoBERTa model, chosen for its strong linguistic representation capabilities. Key methodological contributions include:
- Data Augmentation: To combat data scarcity and improve the representation of less frequent emotions, the authors augmented the training dataset by automatically translating instances from German and Brazilian Portuguese datasets into English using the DeepL Pro API.
- Class Imbalance Mitigation: A specific strategy was implemented within the loss function (BCEWithLogitsLoss) using a
pos_weightparameter. This weight is adjusted based on the relative frequency of each emotion class, penalizing errors on minority classes more heavily to improve overall model performance and accuracy. - Preprocessing Pipeline: A robust preprocessing pipeline was developed, including Unicode normalization, removal of special characters and digits, lowercasing, and replacement of URLs, mentions, and hashtags with generic tokens. Emojis were also identified and tagged.
- Model Architecture: The core model consists of RoBERTa for contextual embeddings, followed by an attention layer to weigh token importance, and a linear classification layer that outputs logits for each emotion. A mean aggregation mechanism is used to consolidate token representations.
Datasets
- BRIGHTER Dataset: The primary dataset used for this task is derived from BRIGHTER, a multilingual collection for text-based emotion recognition. For this study, the English subset was utilized, comprising 2,768 training instances, 116 development instances, and 2,767 test instances.
- Augmented Dataset: To enrich the training data, instances from German and Brazilian Portuguese subsets of BRIGHTER were translated into English, adding approximately 7,597 instances. The emotion "disgust" was excluded from the English dataset due to its scarce representation.
Key Insights and Ideas
- Data Augmentation is Crucial: The significant increase in data diversity through translation demonstrably improved the model's generalization ability and performance, particularly for underrepresented emotions.
- Class Imbalance is a Major Hurdle: The authors explicitly acknowledge that class imbalance significantly impacts minority class detection, motivating their specific loss function adjustment strategy.
- RoBERTa's Strength: RoBERTa proved to be a strong choice for this task, outperforming BERT, likely due to its training advantages for English text and its ability to capture nuanced linguistic representations.
- Context Matters for Ambiguous Emotions: The model struggles with emotions that heavily rely on context or idiomatic expressions, leading to confusion between similar emotions like fear and sadness, or fear and surprise.
Key Results
The experimental results showed competitive performance:
- Development Set: Macro F1 score of 0.6577, Micro F1 score of 0.6571.
- Test Set: Macro F1 score of 0.6266, Micro F1 score of 0.6787.
- Competition Ranking: The model achieved 70th place in the SemEval-2025 Task 11 competition.
- Emotion-Specific Performance: The model performed better on "joy" and "anger" but struggled with "surprise," "sadness," and "fear," with "Anger" having the lowest F1 score on the test set (0.4950).
Limitations and Future Work
- Limitations:
- The model still struggles with accurately identifying surprise, sadness, and fear.
- False positives in surprise are attributed to confusion with emphatic expressions conveying fear or disbelief.
- False negatives in sadness indicate difficulty in capturing emotional subtext when figurative language is used without explicit terms.
- Ambiguous phrases where emotion depends heavily on context lead to omissions in detecting fear and joy.
- The paper notes that the model's performance indicates variability, suggesting the need for further balancing.
- Future Work:
- Explore advanced class balancing strategies.
- Implement automatic hyperparameter tuning techniques.
- Investigate search methods to optimize hyperparameters across multiple pre-trained models.
- Explore hybrid approaches that integrate complementary architectures.
Summary updated