Samsung Research China-Beijing at SemEval-2024 Task 3: A multi-stage framework for Emotion-Cause Pair Extraction in Conversations

Shen Zhang, Haojie Zhang, Jing Zhang, Xudong Zhang, Yimeng Zhuang, Jinting WuNAACL 2024

TL;DR

This paper introduces a novel multi-stage framework for automatically identifying emotions and their underlying causes in conversations. By combining a powerful LLM for initial emotion recognition with specialized models for cause extraction, the authors achieved top performance in the SemEval-2024 Task 3 competition, demonstrating a significant step forward in nuanced conversational AI.

What Problem Does This Paper Address?

In human-computer interaction, understanding a user's emotions is crucial for creating more natural and empathetic AI. However, simply identifying an emotion isn't enough; knowing why someone feels a certain way is key to truly understanding and responding appropriately. This is particularly challenging in conversations, where emotions can be subtle, implied, and influenced by a complex interplay of utterances. Existing methods for identifying "emotion-cause pairs" (i.e., an emotion and the specific text that triggered it) have struggled with performance, often treating emotion recognition and cause extraction as separate, potentially limiting, tasks. This paper tackles the challenge of accurately and comprehensively extracting these emotion-cause pairs from conversational text, aiming to bridge the gap between simple emotion detection and deep emotional understanding.

Methods

The authors propose a three-stage pipeline framework:

  1. Emotion Recognition (InstructERC): They leverage the Llama-2-based InstructERC model, which reformulates emotion recognition as a generative task. This model uses a prompt template that includes job descriptions, historical utterance context, and a set of emotion labels. To further enhance performance, they incorporate auxiliary tasks like sub-label recognition, positive/negative recognition, and even visual cues (like background descriptions and speaker movements) when available.
  2. Causal Pair Extraction (TSAM Model): For identifying the emotion-cause pairs, they employ the TSAM (Two-Stream Attention Model). TSAM integrates speaker and emotion information through separate attention networks (SAN and EAN) and then uses a mutual Bi-Affine transformation (Interaction Network) to exchange information between them. This allows for a more nuanced understanding of how utterances and emotions relate.
  3. Causal Span Extraction (MuTEC): To pinpoint the exact textual span within a cause utterance that triggered the emotion, they utilize MuTEC, an end-to-end multi-task learning framework. MuTEC jointly extracts causal pairs and causal spans, treating emotion prediction as an auxiliary task to improve overall performance.

Datasets

The primary dataset used is ECF (Emotion-Cause-in-Friends), a multimodal dataset containing 1,344 conversations with 13,509 utterances and 9,272 annotated emotion-cause pairs. This dataset includes textual, visual, and acoustic modalities. The paper also mentions using the SHARK dataset for splitting the ECF data into training, validation, and test sets.

Key Insights and Ideas

  • Pipeline Power: A multi-stage pipeline, where each stage is optimized for a specific sub-task (emotion recognition, pair extraction, span extraction), proves more effective than end-to-end approaches for this complex problem.
  • LLMs for Emotion Foundation: Using a powerful LLM like Llama-2 for the initial emotion recognition stage provides a strong, context-aware foundation for subsequent cause extraction.
  • Auxiliary Tasks Boost Performance: Incorporating auxiliary tasks (like sub-label recognition, speaker identification, and even visual descriptions) significantly improves the performance of the core emotion recognition model.
  • Multimodality's Potential (and Challenges): While audio and visual features show promise, their integration into pure textual models can sometimes lead to performance degradation, suggesting the need for more sophisticated fusion strategies.
  • Hierarchical Emotion Labels: Organizing emotion labels hierarchically (neutral, positive, negative, with sub-categories) can aid in more granular emotion recognition.

Key Results

The proposed framework achieved first place in both subtasks of SemEval-2024 Task 3.

  • Emotion Recognition (ERC): The best weighted average F1 score achieved was 58.64, using Llama-2-13B with historical clips descriptions. This represents a significant improvement over baseline InstructERC models.
  • Causal Emotion Entailment (CEE) / Causal Pair Extraction: The model achieved a weighted average F1 score of 38.7 on the official evaluation dataset, significantly outperforming the second-place participant's score of 26.40.
  • Causal Span Extraction (CSE): The model achieved a weighted average proportional F1 score of 32.23, also surpassing the second-place participant.

Limitations and Future Work

  • Error Propagation: A limitation of the pipeline approach is that errors in earlier stages (like emotion recognition) can propagate to later stages.
  • Multimodal Integration: While the paper explored multimodal features (audio, vision), the results showed a decreasing trend compared to pure text. The authors suggest that a more "reasonable approach to incorporate multi-modality" is needed for future work.
  • Future Directions: The authors plan to explore further integration of audio and visual modalities to enhance performance. They also aim to develop more sophisticated methods for multimodal fusion in conversation analysis.

Summary updated

BibTeX
@inproceedings{zhang2024samsung,
  author = {Shen Zhang and Haojie Zhang and Jing Zhang and Xudong Zhang and Yimeng Zhuang and Jinting Wu},
  title = {{Samsung Research China-Beijing at SemEval-2024 Task 3: A multi-stage framework for Emotion-Cause Pair Extraction in Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure