TransMistral at SemEval-2024 Task 10: Using Mistral 7B for Emotion Discovery and Reasoning its Flip in Conversation

Marco SiinoNAACL 2024

TL;DR

This paper explores using the Mistral 7B language model for emotion recognition in code-mixed Hindi-English conversations. By translating conversations to English and employing a few-shot learning approach with Mistral 7B, the authors achieved a respectable F1 score, demonstrating the potential of large language models for this challenging task.

What Problem Does This Paper Address?

Emotion Recognition in Conversation (ERC) is a crucial area in affective computing, aiming to understand the emotional state of speakers within dialogues. This is vital for applications like building better dialogue systems and analyzing social sentiment. However, accurately recognizing emotions in conversations, especially those that are code-mixed (like Hindi and English mixed together), presents significant challenges. These include understanding nuanced emotional expressions, handling the complexities of mixed languages, and dealing with the dynamic nature of emotions that can shift within a conversation. The EDiReF shared task at SemEval-2024 specifically targets these issues, focusing on both emotion recognition and the reasoning behind emotion "flips" (changes in emotion).

Methods

The authors tackled the Emotion Recognition in Conversation (ERC) subtask using a two-pronged approach:

  1. Translation: Code-mixed Hindi-English conversations were translated into English using the deep_translator library with Google Translator. This step aims to simplify the input for the language model by standardizing the language.
  2. Few-Shot Learning with Mistral 7B: The translated conversations were then fed into the Mistral 7B language model. Mistral 7B, a powerful 7-billion parameter model known for its efficiency and performance, was used in a few-shot learning setup. This involved providing the model with a small number of labeled examples (around 80 from the training set and 80 from the development set) as context, along with the definitions of the possible emotions. The model was then prompted to predict the emotion for a given utterance from the test set. The authors explicitly mention that they did not perform any additional preprocessing on the text, as their research suggested it has minimal impact on Transformer-based models.

Datasets

The experiments were conducted using datasets from the SemEval-2024 Task 10 (EDiReF). This task includes:

  • ERC in Hindi-English code-mixed conversations: This is the primary focus of the paper.
  • Emotion Flip Reasoning (EFR) in Hindi-English code-mixed conversations.
  • EFR in English conversations.

The authors utilized labeled training and development sets for their few-shot learning approach and evaluated their model on the official test set.

Key Insights and Ideas

  • Translation as a Simplifier: Translating code-mixed conversations to a single language (English in this case) can be an effective strategy to leverage powerful monolingual language models for tasks involving mixed-language data.
  • Mistral 7B's Few-Shot Capabilities: Mistral 7B, even without extensive fine-tuning, demonstrates promising performance in recognizing emotions in conversations when provided with a few relevant examples. This highlights the power of in-context learning for LLMs.
  • Minimal Preprocessing for Transformers: The paper reinforces the idea that for Transformer models, extensive text preprocessing might not always yield significant improvements and can sometimes be omitted to maintain efficiency.
  • Potential for LLMs in Affective Computing: The results suggest that large language models are well-suited for complex natural language understanding tasks like emotion recognition, even in challenging scenarios like code-mixed dialogues.

Key Results

The proposed approach, "TransMistral 7B," achieved a Weighted F1 score of 0.36 on the official test set for Task 1 (ERC for code-mixed conversations). While this score placed them twenty-second in the final ranking, the authors note that their method required no additional pre-training and was computationally feasible, utilizing free resources like Google Colab. They also reported scores for Task 2 (EFR for code-mixed) and Task 3 (EFR for English) as 0.10 and 0.22 respectively, though they did not actively participate in these subtasks.

Limitations and Future Work

The authors acknowledge that their approach has "further room for improvements," as indicated by their final ranking. They suggest several potential avenues for future work:

  • Leveraging Zero-Shot Capabilities: Exploring the zero-shot capabilities of other models like GPT and T5.
  • Expanding Training Data: Increasing the size of the training set by incorporating additional data.
  • Integrating Domain Knowledge: Adopting alternative strategies for integrating ontology-based domain knowledge.
  • Fine-tuning and Recontextualization: Fine-tuning the model or recontextualizing the problem as a more traditional text classification task.
  • Exploring Other Few-Shot/Data Augmentation Techniques: Investigating other few-shot learning or data augmentation strategies that have shown success in related tasks.

Summary updated

BibTeX
@inproceedings{siino2024transmistral,
  author = {Marco Siino},
  title = {{TransMistral at SemEval-2024 Task 10: Using Mistral 7B for Emotion Discovery and Reasoning its Flip in Conversation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure