PWEITINLP at SemEval-2024 Task 3: Two Step Emotion Cause Analysis

Sofiia Levchenko, Rafał Wolert, Piotr AndruszkiewiczNAACL 2024

TL;DR

This paper tackles the challenging task of identifying emotion-cause pairs within conversations. The authors propose a novel two-step approach that leverages GPT-3 for emotion classification and SpanBERT for extracting the specific text segments that cause those emotions, achieving a 9th place ranking in the SemEval-2024 competition.

What Problem Does This Paper Address?

Understanding the nuances of human emotion in text is crucial for many applications, from sentiment analysis to building more empathetic AI. However, simply identifying an emotion in a piece of text is often insufficient. To truly grasp the emotional context, we need to know why that emotion is being expressed. This is the core of the Emotion Cause Pair Extraction (ECPE) task. Existing methods for ECPE have faced challenges, including the difficulty of accurately linking emotions to their specific textual causes and the potential for error propagation in multi-step approaches. This paper addresses these shortcomings by proposing a refined two-step framework for ECPE in conversational settings.

Methods

The authors adopt a two-step strategy for ECPE:

  1. Emotion Classification (Subtask 1.1): They utilize GPT-3, a powerful large language model, for this task. Two configurations were explored: a standard GPT-3 Assistant and one enhanced with a knowledge retrieval functionality, where specific labeled data was provided as a knowledge base. This allowed GPT-3 to access relevant examples for better prediction.
  2. Emotion-Cause Pair Extraction (Subtask 1.2): For this step, the authors fine-tune the SpanBERT model. SpanBERT is well-suited for question-answering tasks, and the authors frame emotion-cause extraction as a Q&A problem. Specifically, they formulate questions like "What caused the [predicted emotion]?" and use SpanBERT to extract the answer (the cause utterance) from the conversation context.

This two-step approach, while acknowledging the potential for error propagation, aims to leverage the strengths of each model for its respective task.

Datasets

The primary dataset used in this work is the SemEval-2024 Task 3 dataset for Multimodal Emotion Cause Analysis in Conversations. This dataset contains conversations with annotated utterances, speaker information, and expressed emotions. For Subtask 1.1 (emotion classification), the dataset was transformed into a format suitable for GPT-3, with each utterance paired with its conversational context. For Subtask 1.2 (cause extraction), the dataset was further adapted for a question-answering format, including questions about the cause of a predicted emotion and the corresponding answer spans. The paper also mentions using a portion of the training data to enrich the GPT-3 model's knowledge base.

Key Insights and Ideas

  • GPT-3 with Knowledge Retrieval Enhances Emotion Classification: Providing GPT-3 with a curated knowledge base significantly improved its performance in emotion classification, leading to better accuracy and F1 scores compared to a standalone GPT-3 model. This highlights the value of targeted data augmentation for LLMs.
  • Framing Cause Extraction as Q&A is Effective: Using SpanBERT for question-answering to extract cause utterances proved to be a viable strategy, demonstrating the versatility of Q&A models beyond traditional question-answering scenarios.
  • Two-Step Approach Balances Complexity and Performance: While aware of potential error propagation, the authors found that separating the tasks allowed for specialized models (GPT-3 for classification, SpanBERT for extraction) to perform well, ultimately contributing to their competitive ranking.

Key Results

The paper reports the following key results:

  • Subtask 1.1 (Emotion Classification): The GPT-3 based Assistant with a knowledge base achieved a weighted F1-score of 0.52, with precision and recall also at 0.55 and 0.52, respectively. This outperformed the GPT-3 model without a knowledge base.
  • Subtask 1.2 (Emotion-Cause Pair Extraction): The SpanBERT model, fine-tuned for the Q&A task, achieved an Exact Match (EM) score of 21.42 and an F1-score of 33.87.
  • Competition Performance: The overall system, combining these two subtasks, achieved 9th place out of 29 teams in the SemEval-2024 Task 3 competition, with weighted average F1 scores around 0.07.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Textual Data Limitations: Determining expressed emotion solely from text can be challenging due to the absence of non-verbal cues.
  • Model Hallucinations: Despite precise instructions, models can sometimes "hallucinate" or provide unsuitable answers, misinterpreting emotions.
  • Dataset Balancing and Preprocessing: More attention is needed for preprocessing and analyzing the train, validation, and test datasets to ensure more meaningful and balanced questions and answers. Larger datasets would also be beneficial for fully utilizing fine-tuning.
  • Hyperparameter Tuning: More thorough hyperparameter tuning could further optimize model performance.
  • Future Work: The authors suggest exploring GPT-4 architecture, experimenting with different span extraction model architectures, and improving the overall score by addressing the limitations mentioned above. They also note that while BiLSTM models were explored for classification, their performance was limited compared to GPT-3, especially with unknown words and short sentences, suggesting that more advanced context-aware techniques are needed for such models.

Summary updated

BibTeX
@inproceedings{levchenko2024pweitinlp,
  author = {Sofiia Levchenko and Rafał Wolert and Piotr Andruszkiewicz},
  title = {{PWEITINLP at SemEval-2024 Task 3: Two Step Emotion Cause Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure