PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations

Roman Kazakov, Kseniia Petukhova, Ekaterina KochmarNAACL 2024

TL;DR

This paper presents PetKaz, a system for extracting emotion-cause pairs from conversations, achieving 2nd place in SemEval-2024 Task 3. Their approach combines a fine-tuned LLM for emotion classification with a BiLSTM network for cause extraction, demonstrating strong performance on a challenging multimodal dataset.

What Problem Does This Paper Address?

Modern dialogue systems, even those powered by advanced LLMs, often struggle to exhibit genuine human-like understanding of emotions and their underlying causes within conversations. While LLMs can be trained to avoid harmful outputs, they often lack the nuanced ability to grasp the emotional dynamics that drive human interaction. This gap hinders the development of truly empathetic and context-aware conversational agents. The paper tackles the problem of identifying not just the emotion expressed in an utterance, but also the specific utterance(s) that caused it, a crucial step towards more sophisticated dialogue understanding.

Methods

The PetKaz system employs a two-stage pipeline:

  1. Emotion Classification: A fine-tuned GPT-3.5 LLM is used to classify utterances into emotion categories (including neutral). The model is prompted with the preceding utterance and the target utterance. This stage was crucial, as zero-shot and few-shot approaches performed poorly on the dataset.
  2. Cause Extraction: A BiLSTM-based neural network is used to identify causal utterances. Utterance embeddings from a pre-trained BERT model are enriched with context using a BiLSTM. These enriched embeddings, along with speaker vectors and emotion labels, are fed into a feed-forward network for binary classification of potential causal utterances. Notably, the authors chose to extract the entire causal utterance rather than specific spans, citing inconsistencies and ambiguity in the dataset's span annotations.

Datasets

The primary dataset used is the Emotion-Cause-in-Friends (ECF) dataset, which is derived from the Friends TV series and annotated with emotion-cause pairs. This dataset is specifically designed for the SemEval-2024 Task 3 (ECAC) and includes emotions like anger, disgust, fear, joy, sadness, surprise, and neutral. The authors created a development set by splitting the provided training data in a 9:1 ratio.

Key Insights and Ideas

  • LLM Fine-tuning is Key: For this specific task and dataset, fine-tuning a powerful LLM like GPT-3.5 significantly outperformed zero-shot and few-shot approaches for emotion classification. This highlights the importance of task-specific adaptation for LLMs.
  • Span Extraction Challenges: The authors' decision to extract whole utterances instead of specific spans for causes is a significant insight. It suggests that the dataset's span annotations might be noisy or inconsistently applied, making whole-utterance extraction a more robust strategy in this context. This also points to a potential area for dataset improvement.
  • Neutral Utterances are Tricky: The model frequently misclassified neutral utterances, often mistaking greetings or expressions of gratitude for joy. This indicates that subtle cues and context beyond explicit text are vital for accurate emotion detection, even for seemingly simple emotions.
  • Cause-Emotion Distance Matters: The model's performance in identifying causes was negatively correlated with the distance between the causal utterance and the emotional utterance. This aligns with the intuition that closer utterances are more likely to be direct causes.

Key Results

The PetKaz system achieved a weighted-average proportional F₁ score of 0.264 on the test set, securing 2nd place out of 15 participating teams in Subtask 1 of SemEval-2024 Task 3. The system's weighted-average strict F₁ score was 0.104. The fine-tuned GPT-3.5 model achieved an F₁ score of 0.64 for emotion classification, outperforming zero-shot (0.54) and few-shot (0.51) approaches.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Data Annotation: The dataset's annotation quality, particularly for causal spans, presents challenges. Future work could involve improving annotation consistency or using LLMs for more robust annotation.
  • Speaker Representations: Enhancing speaker representations could further improve dialog understanding.
  • LLM-based Cause Extraction: Developing more sophisticated LLM-based methods for cause extraction could yield better results.
  • Model Accessibility: Due to OpenAI's policies, the fine-tuned model cannot be shared, requiring users to perform their own fine-tuning. Making the system fully open-source or using proprietary LLMs with more specific frameworks is a potential future direction.
  • Domain and Language Specificity: The current model is trained on a single domain (Friends series) and the English language. Expanding to other domains and languages would be necessary for broader applicability.

Summary updated

BibTeX
@inproceedings{kazakov2024petkaz,
  author = {Roman Kazakov and Kseniia Petukhova and Ekaterina Kochmar},
  title = {{PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure