UIR-ISC at SemEval-2024 Task 3: Textual Emotion-Cause Pair Extraction in Conversations

Hongyu Guo, Xueyao Zhang, Yiyang Chen, Lin Deng, Binyang LiSemEval @ NAACL 2024 · OpenReview

Abstract

The goal of Emotion Cause Pair Extraction (ECPE) is to explore the causes of emotion changes and what causes a certain emotion. This paper proposes a three-step learning approach for the task of Textual Emotion-Cause Pair Extraction in Conversations in SemEval-2024 Task 3, named ECSP. We firstly perform data preprocessing operations on the original dataset to construct negative samples. Secondly, we use a pre-trained model to construct token sequence representations with contextual information to obtain emotion prediction. Thirdly, we regard the textual emotion-cause pair extraction task as a machine reading comprehension task, and fine-tune two pre-trained models, RoBERTa and SpanBERT. Our results have achieved good results in the official rankings, ranking 3rd under the strict match with the Strict F1-score of 15.18%, which further shows that our system has a robust performance.

Introduction

In Natural Language Processing (NLP), identifying what someone is feeling (Emotion Recognition) is a well-studied field. However, understanding why they feel that way—Emotion Cause Pair Extraction (ECPE)—is much more challenging, especially in conversations.

Unlike static documents, conversations involve multiple speakers, shifting contexts, and "causal spans" (specific snippets of text that trigger an emotion) that might appear several utterances before the emotion is actually expressed. The authors tackle Subtask 1 of SemEval-2024 Task 3, which requires models to not only label an emotion but also pinpoint the exact text fragment in the dialogue history that caused it.

Methodology: The ECSP Framework

The authors propose a pipeline called Emotion-Cause-Span Pair Extraction in Conversation (ECSP), which consists of three distinct modules as shown in the architecture below:

The ECSP framework architecture, illustrating the pipeline from data preprocessing to MRC-based cause extraction.
  1. Data Preprocessing: Since the original dataset only contains positive examples (utterances with known causes), the team constructed a large set of negative samples. They paired emotional utterances with every preceding utterance in the conversation history that wasn't a labeled cause to help the model learn what a "non-cause" looks like.
  2. Emotion Classification: They used a pre-trained BERT model to generate semantic representations of utterances. These were fed into a Feed-Forward Neural Network (FFNN) to predict the specific emotion (e.g., joy, anger, surprise).
  3. Textual Emotion-Cause Pair Extraction: This is the core innovation. They treated the extraction of the causal text as a Machine Reading Comprehension (MRC) task.
  4. Context: The conversation history.
  5. Question: A natural language prompt: "The emotion utterance is <Ui>. What is the causal span from the context that causes the emotion <Ei>?"
    • Answer: The model (fine-tuned RoBERTa or SpanBERT) predicts the start and end indices of the causal text within the context.

To handle the massive imbalance between "cause" and "non-cause" samples, they employed Focal Loss, which forces the model to focus more on "hard" samples that are difficult to classify rather than getting overwhelmed by easy negative examples.

Dataset and Experimental Setup

The study utilized the ECF (Emotion-Cause-Span in Forecasting) dataset provided by SemEval-2024 Task 3. The dataset consists of 1,344 conversations and 13,509 utterances. Over 55% of utterances are labeled with emotions, and 91.34% of those have a corresponding cause. A key complexity of this dataset is that a single emotion can be triggered by multiple causal utterances across the dialogue.

Key Results and Discussion

The system was evaluated using two metrics: Strict Match (the predicted span must be identical to the ground truth) and Proportional Match (partial credit for overlapping spans).

Performance comparison of RoBERTa and SpanBERT models under strict and proportional matching criteria, highlighting the impact of context.
  • Official Ranking: The team ranked 3rd in the Strict Match category.
  • Best Performance: Their SpanBERT model with contextual information achieved a Strict F1-score of 15.18% and a Proportional F1-score of 19.63%.
  • The Power of Framing: By turning an extraction task into a Q&A/MRC task, the authors could leverage models like SpanBERT, which are specifically pre-trained to identify spans of text. This proved more effective than traditional sequence labeling.
  • Context is Crucial: The authors found that adding the "context of the whole dialogue" significantly boosted performance. For the RoBERTa model, adding context increased the Strict F1-score by 1.06% and the Proportional F1-score by 1.36%.
  • SpanBERT vs. RoBERTa: In almost every test scenario, SpanBERT outperformed RoBERTa. This suggests that SpanBERT’s unique pre-training objective (predicting masked spans of text) is highly compatible with identifying causal triggers in dialogue.

Conclusion

The authors acknowledge that extracting the exact textual span remains incredibly difficult. Even the top-performing models in the competition struggle to reach high F1 scores, indicating that the nuance of human emotion-causality is still hard for current LLMs to pin down perfectly. Future work will focus on investigating speaker dynamics and integrating external knowledge bases to help the model understand common-sense causal relationships that aren't explicitly stated in the text.

Summary updated

Abstract

The goal of Emotion Cause Pair Extraction (ECPE) is to explore the causes of emotion changes and what causes a certain emotion. This paper proposes a three-step learning approach for the task of Textual Emotion-Cause Pair Extraction in Conversations in SemEval-2024 Task 3, named ECSP. We firstly perform data preprocessing operations on the original dataset to construct negative samples. Secondly, we use a pre-trained model to construct token sequence representations with contextual information to obtain emotion prediction. Thirdly, we regard the textual emotion-cause pair extraction task as a machine reading comprehension task, and fine-tune two pre-trained models, RoBERTa and SpanBERT. Our results have achieved good results in the official rankings, ranking 3rd under the strict match with the Strict F1-score of 15.18%, which further shows that our system has a robust performance.
BibTeX
@inproceedings{guo2024uir,
  author = {Hongyu Guo and Xueyao Zhang and Yiyang Chen and Lin Deng and Binyang Li},
  title = {{UIR-ISC at SemEval-2024 Task 3: Textual Emotion-Cause Pair Extraction in Conversations}},
  booktitle = {SemEval @ NAACL},
  year = {2024},
  url = {https://openreview.net/forum?id=p1VlSyeRTj},
}
Figure