TL;DR
This paper presents a multimodal approach for identifying the causes of emotions in conversations, leveraging text, audio, and video. The authors achieved 6th place in the SemEval-2024 Task 3 competition by fusing these modalities and jointly predicting emotion-cause links and the target emotion.
What Problem Does This Paper Address?
Understanding the causes of emotions in human interactions is a complex but crucial task for building more empathetic AI systems. While previous work focused on text-only analysis, natural conversations are inherently multimodal, incorporating nuances from tone of voice, facial expressions, and contextual events. This paper tackles the challenge of Multimodal Emotion Cause Analysis in Conversations (ECPE), specifically aiming to extract pairs of (emotion, cause) within dialogues, considering all available modalities. The task is split into two subtasks: one focusing solely on text for cause span identification, and the other requiring multimodal analysis to identify emotion-cause pairs.
Methods
The UWBA team developed a system that processes text, audio, and video modalities separately before fusing them.
- Text Encoding: A transformer-based encoder (BERT) processes the entire dialogue to capture context. Utterances are separated by special tokens, and positional encoding is applied. The representation for each utterance is derived by averaging its token embeddings.
- Audio Encoding: Two methods were explored:
- MFCC Feature Extraction: This involves denoising audio, separating the main speaker's voice, and then extracting Mel-frequency cepstral coefficients. These features are fed into a Long Short-Term Memory (LSTM) model trained for emotion recognition.
- Wav2Vec: A pre-trained Wav2Vec model was fine-tuned for emotion classification. The final layer's output is averaged to create a 1024-dimensional utterance representation.
- Video Encoding: A pre-trained ResNeXt 3D-CNN model (depth 101) was used. For every 16 frames, it outputs a 2048-dimensional vector, with temporal pooling used for longer videos.
- Fusion Module: A transformer-based encoder is used to fuse the modality representations. Each modality's vector is first linearly projected to a common dimension. These projected vectors then serve as tokens for the transformer encoder. The fusion happens at the utterance level. Several fusion strategies were explored, including aggregation functions (Avg, Max, Min) and the use of a learnable fusion token (FT).
- Joint Prediction: The fused utterance representations are used for both emotion prediction and link prediction (identifying the cause-emotion pair). These two tasks are trained simultaneously using a combined loss function, hypothesizing that emotions and links can positively influence each other.
- Text Span Classification (Subtask 1): For identifying the specific text segment causing the emotion, a transformer model is used. It predicts one of five categories (Whole Utterance, First Part, Middle Part, Last Part, Other) based on individual utterances, with regular expressions used to extract the actual span.
Datasets
The primary dataset used is the Emotion-Cause-in-Friends dataset (Wang et al., 2022), which consists of recordings from the sitcom Friends and is annotated for emotion-cause pairs.
Key Insights and Ideas
- Text is King (Mostly): For emotion detection, text-only models performed comparably to multimodal ones, suggesting that the core emotional information is heavily present in the dialogue transcriptions. Audio and video provided only a small positive impact for link prediction.
- Dialogue Context Matters for Links: Unlike emotion detection, the context of the entire dialogue was crucial for effective link prediction.
- Fusion Strategy Nuances: The best fusion strategy differed between tasks. For emotion prediction, a text-only approach or a simple fusion token (SingleFT/MainSpeakerFT) worked best. For link prediction, aggregation-based fusion strategies (like MaxFusion) proved more beneficial.
- Joint Training Benefits: Training emotion and link prediction jointly, rather than separately, led to better overall results, indicating a synergistic relationship between understanding emotions and their causes.
- Text Span Annotation Challenges: The analysis revealed that a significant portion of the training data lacked specific text spans for causes, often defaulting to "Whole Utterance." This highlights the difficulty in precisely annotating cause spans, even for human annotators.
Key Results
- The UWBA system achieved 6th place overall in the SemEval-2024 Task 3 competition for both subtasks.
- For Subtask 2 (Multimodal ECPE), their combined model (best emotion detection + best link prediction) achieved a weighted F1 score of 0.331 on the development dataset.
- For Subtask 1 (Text Span Classification), their best model achieved a weighted-average proportional F1 score of 0.208 on the test data.
- The best performing models for emotion detection were often text-only, achieving accuracies around 76.8% and Macro F1 scores around 63.1% (Table 4).
- Link prediction saw improvements with multi-task training, reaching a Jaccard index of 0.359.
Limitations and Future Work
- Modality Impact: The paper notes that audio/video modalities had a limited positive impact, especially for emotion detection. Future work could explore better preprocessing for these modalities, such as focusing on the main speaker's face in videos.
- Model Size: The authors suggest that using larger models could potentially improve performance.
- Separate vs. Joint Training: The current submission uses a composition of two separate models (one for emotion, one for link prediction). Future work aims to develop a single learning objective that manages both tasks, potentially simplifying the training and avoiding the prediction of neutral links. This could involve a link module with a multi-head self-attention matrix where each head represents a specific emotion link.
- Fusion Strategy Conflicts: The observed conflicts in optimal fusion strategies between tasks suggest that a one-size-fits-all approach might not be ideal.
Summary updated