Chinchunmei at SemEval-2025 Task 11: Boosting the Large Language Model's Capability of Emotion Perception using Contrastive Learning

Tian Li, Yujian Sun, Huizhi Liang

Abstract

Text-Based Emotion Detection (TBED) remains a significant challenge in natural language processing due to the subtle, culturally-dependent nature of emotional expression. This paper presents the Chinchunmei system developed for SemEval-2025 Task 11, which enhances the emotion perception capabilities of LLaMa3-8B through contrastive learning. We evaluate three training strategies: standard supervised fine-tuning, sample-based contrastive reasoning calibration (CRC), and generation-based preference optimization (DPO/SimPO). Our results indicate that while Direct Preference Optimization (DPO) effectively captures emotional intensity, multilingual training can introduce cultural noise that degrades performance in specific languages like English. The system achieved a top-10 ranking in 16 languages for classification and secured 6th place in English emotion intensity prediction.

Introduction

Text-Based Emotion Detection (TBED) involves identifying complex emotional states and their intensities from text. Unlike basic sentiment analysis, TBED must navigate metaphors, situational cues, and diverse cultural contexts. The SemEval-2025 Task 11 competition, "Bridging the Gap in Text-Based Emotion Detection," introduced two primary hurdles: multilinguality across 28 languages and granularity in predicting intensity across six categories (anger, fear, joy, sadness, surprise, and disgust). This study investigates whether contrastive learning—teaching models by comparing "good" and "bad" examples—can help Large Language Models (LLMs) navigate these nuances more effectively than standard fine-tuning.

Methodology

The Chinchunmei system utilizes LLaMa3-Instruct-8B as its backbone. The researchers explored three distinct training paradigms to optimize emotion perception:

  1. Standard Prediction (SP): A baseline approach using standard Supervised Fine-Tuning (SFT) to map text directly to emotion labels.
  2. Sample-Based Contrastive Learning (CRC): This method, Contrastive Reasoning Calibration, trains the model to compare pairs of text samples and explain their relative emotional intensities. During inference, a voting mechanism aggregates multiple comparisons to determine the final label.
  3. Generation-Based Contrastive Learning (DPO & SimPO): These preference optimization techniques involve creating "mutated" (incorrect) labels for training data. The model is then trained to prefer the original, correct label over the mutated one. Direct Preference Optimization (DPO) uses a reference model for stability, while Simple Preference Optimization (SimPO) is a reference-free alternative.
Overview of the three training strategies: Standard Prediction, Sample-based Contrastive Learning, and Generation-based Contrastive Learning.

Experimental Setup and Datasets

The study utilized the SemEval-2025 Task 11 benchmark, which includes Track A (multi-label classification for 28 languages) and Track B (emotion intensity prediction on a 0-3 scale for 11 languages). The dataset follows Ekman’s framework of six basic emotions. A notable challenge was the absence of certain labels (e.g., "disgust") in specific language subsets like English, requiring the model to maintain flexibility across varying label sets.

Results and Analysis

The Chinchunmei system demonstrated strong competitive performance, particularly in intensity prediction.

Performance comparison of different training strategies on Track A (English).

Key Findings:

  • The Multilingual Penalty: Surprisingly, incorporating non-English data into the training set degraded performance on English tasks. This suggests that emotion perception is highly culture-specific; "averaging" across 28 languages likely introduced conflicting signals that confused the model's perception of English nuances.
  • DPO vs. SimPO: DPO proved more effective for maintaining structured output. While SimPO is memory-efficient, it often caused the model to "forget" correct output formatting. In the intensity prediction task (Track B), the DPO-tuned model outperformed the standard baseline in almost every metric, particularly for sadness and fear.
  • CRC Performance: While CRC struggled overall due to ambiguity in borderline labels, it achieved the highest score for the "Anger" label in classification, suggesting that comparison-based reasoning may be better suited for high-arousal emotions.

Conclusion and Future Work

The Chinchunmei system highlights the potential of preference optimization for fine-grained emotion detection. However, the "multilingual penalty" and the sensitivity of contrastive methods to label ambiguity remain significant hurdles. Future research will focus on using cultural adapters to handle diverse linguistic contexts without cross-lingual interference and refining mutation strategies for more robust contrastive learning. Additionally, the team identified a "development set pitfall," emphasizing the need for larger validation sets to ensure the generalizability of performance gains.

Summary updated

BibTeX
@inproceedings{li2025chinchunmei,
  author = {Tian Li and Yujian Sun and Huizhi Liang},
  title = {{Chinchunmei at SemEval-2025 Task 11: Boosting the Large Language Model's Capability of Emotion Perception using Contrastive Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure