Abstract
We present SemEval-2024 Task 10, a shared task centered on identifying emotions and finding the rationale behind their flips within monolingual English and Hindi-English code-mixed dialogues. This task comprises three distinct subtasks: emotion recognition in conversation for code-mixed dialogues, emotion flip reasoning for code-mixed dialogues, and emotion flip reasoning for English dialogues. Participating systems were tasked to automatically execute one or more of these subtasks. The datasets for these tasks comprise manually annotated conversations focusing on emotions and triggers for emotion shifts. A total of 84 participants engaged in this task, with the most adept systems attaining F1-scores of 0.70, 0.79, and 0.76 for the respective subtasks. This paper summarizes the results and findings from 24 teams alongside their system descriptions.
Introduction
While Artificial Intelligence has made significant strides in Emotion Recognition in Conversation (ERC)—the ability to identify what a speaker is feeling—understanding the causal factors behind emotional shifts remains a complex challenge. SemEval-2024 Task 10, titled "Emotion Discovery and Reasoning its Flip in Conversation" (EDiReF), was designed to push the boundaries of affective computing by introducing Emotion Flip Reasoning (EFR). This task requires models to not only identify emotional states but also pinpoint the specific "triggers" or utterances that cause a speaker's emotion to change (an "emotion-flip").
Problem Statement
The EDiReF task addresses two primary gaps in current Natural Language Processing (NLP) research:
- Causality in Emotion: Moving beyond simple classification to identify the specific prior utterances that precipitate a change in emotional state.
- Linguistic Diversity: Addressing the complexities of "code-mixed" conversations, specifically Hindi-English (Hinglish), where speakers fluidly switch between languages. This is a common real-world phenomenon that poses significant challenges for standard monolingual models.
Task Framework
The competition was structured into three distinct subtasks:
- Subtask A: Emotion Recognition in Hindi-English code-mixed dialogues (8 emotion labels).
- Subtask B: Emotion Flip Reasoning in Hindi-English code-mixed dialogues (identifying triggers for emotional shifts).
- Subtask C: Emotion Flip Reasoning in monolingual English dialogues.
Methodology and Participating Systems
Participating teams employed a diverse array of techniques, ranging from state-of-the-art Large Language Models (LLMs) to classical statistical methods:
- Large Language Models: Models such as BERT, RoBERTa, Mistral-7B, and Zephyr were widely utilized. Many teams employed Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA (Low-Rank Adaptation) or instruction tuning to adapt these models to dialogue-specific contexts.
- Hybrid and Two-Phase Strategies: The top-performing team for Subtask A utilized a hybrid approach, extracting contextual features with DistilBERT and subsequently processing them through classical machine learning classifiers like Support Vector Machines (SVMs) or Random Forests.
- Cross-Lingual Strategies: To manage code-mixing, some participants utilized translation-based pipelines (translating Hindi to English), while others leveraged inherently multilingual models like mBERT.
- Heuristic and Rule-Based Methods: For the reasoning tasks (B and C), several teams found that simple TF-IDF or rule-based methods—specifically those prioritizing the immediately preceding utterance—were surprisingly competitive.
Datasets
The task introduced and extended several benchmarks for the research community:
- MELD-FR: An extension of the Multimodal EmotionLines Dataset (MELD), containing 1,161 English dialogues with 8,387 manually annotated trigger utterances.
- E-MASAC & EFR-MASAC: Based on the MaSaC dataset (derived from the Indian television show Sarabhai vs Sarabhai), these provide code-mixed dialogues annotated for both emotional states and flip triggers.
Key Insights and Results
The competition yielded several significant findings regarding the current state of emotion-aware AI:
- The Recency Heuristic: A dominant finding was that in the majority of conversational data, an emotional flip is triggered by the utterance immediately preceding the flip ($i-1$). This explains why simple models focusing on local context often matched the performance of complex LLMs in reasoning tasks.
- Model Scale vs. Efficiency: In the code-mixed ERC task, DistilBERT outperformed larger models like GPT-4. This suggests that for specialized, relatively small datasets (~8,500 utterances), lighter models may be more effective at capturing nuanced patterns without the overfitting risks associated with massive parameter counts.
- The Challenge of Implicit Triggers: Systems struggled significantly when the cause of an emotional shift was not explicitly stated in the text (e.g., an external event or non-verbal cue).
Performance Summary
| Subtask | Top F1-Score | Winning Team |
|---|---|---|
| Subtask A (Code-mixed ERC) | 0.70 | Team IASBS |
| Subtask B (Code-mixed EFR) | 0.79 | Multiple Teams (Tie) |
| Subtask C (English EFR) | 0.76 | Team GAVx |
Conclusion and Future Directions
SemEval-2024 Task 10 highlights the evolving landscape of emotion reasoning. While models are becoming proficient at identifying triggers within immediate conversational context, several hurdles remain:
- Implicit Reasoning: Developing models capable of inferring external context or "reading between the lines" remains an open challenge.
- Data Imbalance: Datasets remain skewed toward "Neutral" and "Joy," leading to lower performance on rarer emotions like "Contempt" or "Disgust."
- Multimodality: Future iterations of this work will likely need to integrate audio and visual cues, as tone of voice and facial expressions are often the primary indicators of an emotional flip.
Summary updated