LyS at SemEval-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing

Ana Ezquerro, David VilaresNAACL 2024

TL;DR

This paper presents an early prototype for an end-to-end multimodal system designed to identify causal emotion links within conversations. By framing the problem as graph-based parsing, the system leverages a multimodal encoder and a graph-based decoder to predict emotions and their causal relationships between utterances, achieving a 7th place ranking in the SemEval-2024 Task 3 for text-only inputs.

What Problem Does This Paper Address?

Understanding emotions in conversations is crucial, but it's not just about identifying what emotion is expressed. The real challenge lies in understanding why that emotion arises and which specific utterance or event triggered it, especially in multi-party dialogues. This is known as Multimodal Emotion Cause Analysis. Existing work often focuses on recognizing emotions per utterance or extracting simple cause-effect pairs. SemEval-2024 Task 3 aimed to bridge this gap by requiring systems to not only identify emotions but also to pinpoint the causal relationships between utterances and the specific textual spans responsible for triggering those emotions. This task is particularly relevant given the recent advancements in multimodal AI, combining text, audio, and visual information.

Methods

The proposed system is built around a two-module architecture: a multimodal encoder and a graph-based decoder.

  • Multimodal Encoder: This component takes the input conversation and generates a rich representation for each utterance. For Subtask 1 (text-only), it uses a BERT-based encoder. For Subtask 2 (multimodal), it extends this by incorporating representations from a Vision Transformer (ViT) for images and wav2vec 2.0 for audio. The system aims for an end-to-end approach, allowing for seamless integration of different modalities without altering the decoder.
  • Graph-Based Decoder: This module takes the utterance embeddings from the encoder and models the cause-effect relations as a directed labeled graph. It uses a biaffine attention mechanism, adapted from dependency parsing, to predict an adjacency matrix representing the probability of a causal link between utterances. Additionally, a "span attention" module is introduced to identify the specific words within a cause utterance that trigger the emotion in the effect utterance.

Datasets

The primary dataset used for this work is the one provided for SemEval-2024 Task 3, which consists of 1375 multi-party English conversations with annotated emotions, cause-effect relations, and trigger spans. The paper mentions using the Facial Emotion Recognition dataset for pre-training the ViT model.

Key Insights and Ideas

  • Graph-based parsing for emotion causality: Framing the complex problem of emotion cause analysis as a graph-based parsing task, similar to syntactic dependency parsing, is a novel and effective approach. This allows for the structured prediction of relationships between utterances.
  • End-to-end multimodal integration: The architecture is designed to seamlessly integrate various modalities (text, audio, video) into a single encoder, with the decoder remaining modality-agnostic. This simplifies the system and allows for flexible experimentation.
  • Span prediction is crucial: The results highlight the significant impact of accurately predicting the specific textual span that triggers an emotion. Removing span prediction led to a notable increase in F-Score, underscoring its importance for a comprehensive solution.
  • Textual encoder fine-tuning is paramount: The experiments show that fine-tuning the textual encoder (BERT) is critical for achieving good performance. Even with multimodal inputs, if the textual encoder is not fine-tuned, performance suffers significantly.

Key Results

  • Subtask 1 (Text-only): The system achieved a validation score of 9.75 and ranked 7th out of 15 participants in the official evaluation with 6.77 points.
  • Subtask 2 (Multimodal): Post-evaluation results showed that combining text and audio yielded the best performance with a weighted F-Score of 23.36. While image data also improved the text-only baseline, it underperformed compared to the audio model. The best multimodal submission to the official competition achieved 15.32 points, ranking 13th out of 18 participants.
  • Span prediction impact: Removing span prediction from the text-only model led to a substantial increase in F-Score (from 11.25 to 20.43), demonstrating its critical role.

Limitations and Future Work

  • Limited preparation time: The authors acknowledge that their participation was constrained by limited preparation time, which prevented exhaustive hyperparameter tuning and k-fold cross-validation.
  • Computational resources: The computational resources available limited their ability to fine-tune the full multimodal encoder, particularly for text and video data. They relied on projecting multimodal inputs to their contextualizations and training the decoder.
  • Image data limitations: For image data, only a fixed number of sampled frames were used. The authors suggest that providing the full sequence of frames could potentially lead to better performance.
  • Future Work: The authors plan to experiment with smaller and distilled models to enable cheaper fine-tuning of the full multimodal model. They also aim to explore more advanced approaches to enhance their system's performance and efficiency.

Summary updated

BibTeX
@inproceedings{ezquerro2024lys,
  author = {Ana Ezquerro and David Vilares},
  title = {{LyS at SemEval-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure