JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models

Arefa, Mohammed Abbas Ansari, Chandni Saxena, Tanvir AhmadNAACL 2024

TL;DR

This paper presents a two-step approach for multimodal emotion cause analysis in conversations, achieving 4th place in SemEval 2024 Task 3. They explore two methods: fine-tuning Llama 2 models and using GPT-3.5 with in-context learning, demonstrating the effectiveness of LLMs in understanding complex conversational dynamics.

What Problem Does This Paper Address?

Human conversations are rich with emotional cues that are often conveyed through multiple modalities like text, audio, and video. Understanding why certain emotions arise (Emotion Cause Analysis or ECA) is crucial for deeper comprehension of communication. While research exists for unimodal ECA, analyzing these complex, multimodal conversations in natural settings presents a significant challenge. SemEval 2024 Task 3 specifically targets this by introducing the "Multimodal Emotion Cause Analysis in Conversations" (ECAC) task, which requires identifying emotion-cause pairs within conversations that include text, audio, and video.

Methods

The authors propose a two-step framework for ECAC:

  1. Emotion Recognition: First, they identify the emotion of each utterance in a conversation.
  2. Cause Prediction: Then, using the identified emotion labels as guidance, they predict the utterances that caused those emotions.

They explore two distinct approaches:

  • Approach 1: Fine-tuned Llama 2: This involves instruction fine-tuning two separate Llama 2-13b models. One model is trained for emotion recognition, and another for cause prediction. They utilize QLoRA for parameter-efficient fine-tuning.
  • Approach 2: In-Context Learning (ICL) with GPT: This approach leverages GPT-3.5's in-context learning capabilities. They use video captions generated by GPT-4V to provide multimodal context. For emotion recognition, they retrieve similar annotated conversations from the training set as demonstrations. For cause prediction, they retrieve context windows with similar emotions and their causes, also explained by GPT-3.5, to guide the GPT-3.5 model.

A notable technique used in both approaches is the incorporation of "self-causes," where an utterance is considered a cause of itself, which significantly boosts performance.

Datasets

The primary dataset used is the Emotion-Cause-in-Friends (ECF) dataset introduced by Wang et al. (2023a). This dataset comprises 13,509 multimodal utterances from the American sitcom Friends, with 9272 emotion-cause pairs. Each utterance includes text, video, and audio. The dataset is noted to be imbalanced, with a large proportion of neutral emotions.

Key Insights and Ideas

  • Two-step approach is effective: Decomposing the complex ECAC task into separate emotion recognition and cause prediction steps significantly improves performance.
  • LLMs excel with context: Both fine-tuned Llama 2 and GPT-3.5 with in-context learning demonstrate strong performance, highlighting the importance of making LLMs "context-aware" for conversational tasks.
  • Self-causes are crucial: Including an utterance as a potential cause of its own emotion ("self-causes") leads to substantial performance gains, reflecting the intuitive idea that what one says can directly trigger their own emotions.
  • Video captions can be noisy: While GPT-4V can generate video captions, directly using them as additional context for the LLM sometimes led to decreased performance due to noise and ambiguity in the captions.
  • ICL outperforms zero-shot: In-context learning with GPT-3.5 significantly outperforms zero-shot approaches, especially when provided with relevant demonstrations.

Key Results

The authors' system achieved 4th place on the official leaderboard for Subtask 2 of SemEval 2024 Task 3, competing against over 25 teams, with a weighted F1 score of 0.2816.

  • Their In-Context-Learning GPT approach was their best submission, ranking 4th.
  • Their Fine-tuned Llama 2 approach ranked 6th.
  • Ablation studies showed that instruction tuning and ICL significantly improved performance over zero-shot Llama.
  • Adding self-causes consistently improved performance for both zero-shot and context-aware scenarios.
  • Incorporating video captions negatively impacted the performance of the in-context learning approach.

Limitations and Future Work

  • Dataset Specificity: The proposed approaches are tailored to the ECF dataset and may not generalize well to other datasets.
  • Resource Constraints: Due to resource limitations, they used a Llama 2-13b model instead of larger variants and QLoRA for fine-tuning. They also used GPT-3.5 instead of GPT-4 for some tasks.
  • LLM Hallucinations: LLMs can sometimes hallucinate or produce unstructured outputs, requiring retries.
  • Future Work: The authors suggest investigating larger Llama 2 models and GPT-4, along with more sophisticated in-context learning techniques, to further improve performance. They also note that their approaches are specific to one dataset and could benefit from generalization studies.

Summary updated

BibTeX
@inproceedings{arefa2024jmi,
  author = {Arefa and Mohammed Abbas Ansari and Chandni Saxena and Tanvir Ahmad},
  title = {{JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure