TL;DR
This paper explores the zero-shot emotion recognition capabilities of GPT-4o Mini, a small yet cost-efficient LLM, for multilingual text. They introduce a novel two-step approach: "Role-Play Rewriting" to generate emotion-specific text variations and "Contrastive Judging" to compare these with the original text for emotion classification. This method offers a plug-and-play solution for low-resource settings without requiring labeled data.
What Problem Does This Paper Address?
Accurately recognizing emotions in text is crucial for many NLP applications, but it's particularly challenging in multilingual contexts due to cultural and linguistic nuances. Existing methods often require extensive labeled data for fine-tuning, which is a significant bottleneck, especially for low-resource languages. SemEval-2025 Task 11 specifically targets this by providing a multilingual dataset across 28 languages, pushing for robust emotion detection systems with limited training data. This paper aims to evaluate how well a smaller, more accessible LLM (GPT-4o Mini) can perform this task in a zero-shot setting, meaning without any task-specific training.
Methods
The core of the proposed method is a two-step pipeline designed for zero-shot inference:
- Role-Play Rewriting: The GPT-4o Mini model is prompted to rewrite a given input sentence multiple times. Each rewrite aims to express a specific target emotion (e.g., joy, sadness, anger) by making minimal changes to the original text, preserving its core meaning. This step essentially asks the LLM to "act out" different emotions.
- Contrastive Judging: In the second step, the model compares the original sentence with each of the rewritten, emotion-specific sentences. It then provides a brief reasoning (using chain-of-thought) for how well the original sentence aligns with each emotion's rewritten version and assigns an intensity score (0-3) for each emotion. This comparative approach helps the model make a more reasoned decision.
This entire process is designed to be "plug-and-play," requiring no fine-tuning or labeled data, and relies on structured output features for parsing results.
Datasets
The evaluation was conducted using the BRIGHTER dataset (Muhammad et al., 2025a), which was provided for SemEval-2025 Task 11. This dataset includes text in 28 languages, ranging from high-resource to low-resource settings. The authors used the development set for prompt iteration and the test set for final evaluation.
Key Insights and Ideas
- Task Decomposition is Key for Small LLMs: The paper strongly suggests that breaking down the complex task of emotion recognition into smaller, manageable steps (rewriting and then judging) significantly enhances the zero-shot capabilities of smaller LLMs like GPT-4o Mini.
- Role-Play as an Explanatory Tool: The "Role-Play Rewriting" step acts as a form of explanation, helping the model clarify the emotional content of ambiguous texts by forcing it to consider how different emotions would manifest.
- Contrastive Judging Aids Reasoning: The "Contrastive Judging" step, by requiring explicit reasoning and comparison, helps the model arrive at a more grounded decision, even if it's prone to occasional misclassifications.
- Low-Resource Languages Remain a Challenge: Performance disparities between high-resource (like English) and low-resource languages (like isiZulu) are significant, highlighting the ongoing challenges in cross-lingual NLP.
- Cost-Effectiveness and Accessibility: The method's reliance on a small LLM and zero-shot learning makes it a highly cost-effective and accessible solution for emotion detection, especially in scenarios where labeled data is scarce.
Key Results
The system achieved near-baseline results on both Track B (Emotion Intensity) and Track C (Cross-lingual Emotion Detection) of SemEval-2025 Task 11.
- Track B (Emotion Intensity): English showed the best performance with an average Pearson correlation of 0.5693. Other languages like Ukrainian (0.3517), Hausa (0.3372), and Chinese (0.3068) followed. Notably, Chinese exhibited the lowest consistency for surprise (r=0.133), indicating struggles with subtle intensities in non-Latin scripts. Low-resource languages like isiZulu (0.1894) and Ukrainian (0.237) lagged behind.
- Track C (Cross-lingual Emotion Detection): Macro-F1 scores varied significantly, with English achieving 0.5598 and isiZulu scoring as low as 0.1894.
Qualitatively, the authors observed that the role-play mechanism helped clarify emotions in ambiguous texts. However, contrastive judging sometimes led to misclassifications, particularly for nuanced distinctions like fear vs. surprise.
Limitations and Future Work
The authors acknowledge that their system's performance was near the baseline and that there's a performance gap between high- and low-resource languages. They also noted that errors in the role-play rewriting step could lead to incorrect judgments.
Future work could involve:
- Refining the Rewriting Step: Exploring alternative ways to generate emotion-specific rewrites or even removing the rewriting stage to isolate the impact of contrastive judging.
- Comparing Across LLMs: Evaluating this approach with different LLMs to understand its generalizability.
- Systematic Ablation Studies: Investigating the contribution of each module (rewriting vs. judging) in isolation.
- Prompt Engineering and Fine-tuning: While the current work is zero-shot, partial fine-tuning or more advanced prompt engineering could potentially improve accuracy, especially across languages.
Summary updated