TL;DR
This paper details the UIMP-Aaman team's participation in SemEval-2025 Task 11, focusing on emotion recognition in multilingual social media and news text. They leverage large language models (LLMs) with in-context learning and fine-tuning to predict both the presence and intensity of emotions across English, German, Spanish, and Portuguese. Their findings highlight the effectiveness of larger LLMs and the varying performance across languages and tasks.
What Problem Does This Paper Address?
The paper tackles the complex challenge of accurately recognizing emotions expressed in text, a crucial aspect of understanding human communication. This task is particularly difficult due to the subjective nature of emotions, the nuances of language, and the need to handle multiple languages. SemEval-2025 Task 11 specifically aims to bridge the gap in human-annotated textual emotion recognition datasets by evaluating models on their ability to detect emotions (Task A) and their intensity (Task B) across different languages, as well as predicting emotions in a target language using a labeled training set from another (Task C).
Methods
The UIMP-Aaman team employed two primary techniques:
- In-context learning: This method involves providing the LLM with a prompt that includes a few examples of the task (e.g., sentence-emotion pairs) without updating the model's weights. The LLM learns from these examples to infer the pattern and perform the task. This approach is "prompting" the model to understand and respond based on the provided context.
- Fine-tuning: This standard technique involves taking a pre-trained LLM and further training it on a specific dataset to adapt its capabilities to a specialized task. The authors fine-tuned a smaller Llama model (3.2-1B) for each language.
For Track A, they experimented with various LLMs, including different versions of Llama and Phi-4, using the HuggingFace transformers library. They also explored the impact of the number of in-context examples on performance. For Tracks B and C, they selected the best-performing model identified in Track A (Llama 3.3-70B) and applied it.
Datasets
The study utilizes the datasets provided for SemEval-2025 Task 11, which cover emotion recognition in English, German, Spanish, and Brazilian Portuguese. The paper does not introduce new datasets but rather focuses on evaluating performance on the existing benchmark.
Key Insights and Ideas
- Larger LLMs generally perform better: Across various configurations, larger models like Llama 3.3-70B consistently outperformed smaller models, especially for tasks involving complex emotion recognition.
- In-context learning is a powerful tool for LLMs: The paper demonstrates that even without fine-tuning, LLMs can achieve competitive results by leveraging well-crafted prompts with relevant examples.
- Language-specific performance variations: The models showed significant differences in performance across languages, with English and Spanish often yielding higher scores. This suggests that language representation and the availability of training data play a crucial role.
- Fine-tuning can be beneficial, but not always superior: While fine-tuning improved performance for some languages in Track A, in-context learning with a large model often matched or surpassed it, especially for English.
- Intensity prediction is more challenging: Task B, which involves predicting emotion intensity, appears to be more difficult, with high-intensity emotions being less frequent in the training data, potentially hindering model learning.
Key Results
- Track A: The Llama 3.3-70B model achieved the best performance, with F1 scores reaching up to 0.77 for Spanish and 0.71 for English. The number of in-context examples also influenced performance, with an optimal number found to be around 40 for some languages.
- Track B: The Llama 3.3-70B model achieved an F1 score of 0.89 for English, 0.67 for German, 0.65 for Portuguese, and 0.66 for Spanish. English showed the highest performance, likely due to its larger representation in the training data.
- Track C: Performance varied depending on the training language used. For predicting English, training with Portuguese yielded the highest F1 score of 0.89.
- Overall Ranking: The proposed LLM solution achieved good results, outperforming the baseline in most tracks and languages, demonstrating the power and convenience of LLMs that do not require extensive training.
Limitations and Future Work
- Computational Resources: The authors acknowledge that they were limited in fine-tuning larger models due to computational and financial constraints, opting to fine-tune a smaller Llama model.
- Data Imbalance: The paper notes that high-intensity emotions are less frequent in the training data for Task B, which can make accurate prediction more challenging.
- Future Work: The authors suggest exploring large LLM fine-tuning with techniques like LoRa (Low-Rank Adaptation) to further improve results. They also imply that further tuning and exploration of different LLMs could yield even better outcomes.
Summary updated