Using Auxiliary Tasks In Multimodal Fusion of Wav2vec 2.0 And Bert for Multimodal Emotion Recognition

Dekai Sun, Yancheng He, Jiqing Han

TL;DR

This paper tackles the challenge of multimodal emotion recognition (MER) by proposing a novel fusion strategy that leverages pre-trained audio (wav2vec 2.0) and text (BERT) models. They introduce two auxiliary tasks to improve how these modalities are integrated, leading to state-of-the-art performance on the IEMOCAP dataset.

What Problem Does This Paper Address?

Multimodal Emotion Recognition (MER) aims to understand human emotions by combining information from various sources like speech, text, and facial expressions. However, two major hurdles exist: a scarcity of labeled data for training robust models, and the inherent difficulty in effectively fusing information from different modalities. Simply concatenating or averaging features often fails to capture the nuanced interplay between modalities that is crucial for accurate emotion detection. This paper seeks to overcome these limitations by proposing a more sophisticated fusion mechanism guided by auxiliary learning objectives.

Methods

The core of the proposed method lies in a modular, end-to-end framework.

  1. Pre-trained Encoders: They utilize pre-trained wav2vec 2.0 for audio and BERT for text to extract rich, contextualized feature representations. These models are then fine-tuned on the MER task.
  2. Fusion Module: A K-layer multi-head cross-attention mechanism is employed to fuse the audio and text features. This allows for dynamic interaction and weighting of information from both modalities.
  3. Auxiliary Tasks: To address the fusion challenge, two novel auxiliary tasks are introduced:
  4. Auxiliary Task 1 (Scrambled Pair Prediction): This task involves randomly shuffling audio and text pairs within a batch and training the model to predict not only the original emotion label but also the combined label of the scrambled pair. This forces the model to learn that each modality contributes to the final fusion, preventing it from relying too heavily on a single modality.
  5. Auxiliary Task 2 (Semantic-Disrupted Alignment): Here, one modality in a pair is replaced with data from a different semantic content but the same emotion category. This encourages the model to focus on emotion-specific features that are aligned across modalities, rather than just semantic content.

Datasets

The primary dataset used for training and evaluation is the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset. This dataset contains improvised and scripted dialogues from 10 actors, covering emotions like happy, excited, angry, sad, neutral, frustrated, and surprise. For their experiments, the authors focused on a subset of these emotions, merging "excited" with "happy" to create a more balanced dataset of 5,531 utterances. They employed a leave-one-session-out 5-fold cross-validation strategy for evaluation.

Key Insights and Ideas

  • Auxiliary tasks are crucial for effective fusion: The paper strongly suggests that simply using powerful pre-trained models and a complex fusion architecture isn't enough. The auxiliary tasks play a vital role in guiding the model to learn how to properly integrate multimodal information.
  • Decoupling fusion from modality dominance: Auxiliary Task 1 is particularly interesting as it forces the model to acknowledge the contribution of both modalities, even when they are artificially recombined. This prevents the model from becoming "lazy" and relying on just one source of information.
  • Focusing on emotion alignment: Auxiliary Task 2 highlights the importance of learning shared emotional representations, even when the explicit content differs. This is key for robust emotion recognition across diverse contexts.
  • The sweet spot for cross-attention layers: The results suggest that a K=2 layer multi-head cross-attention mechanism, combined with the auxiliary tasks, yields the best performance, indicating a balance between capturing complex interactions and avoiding overfitting on a limited dataset.

Key Results

The proposed method achieved state-of-the-art performance on the IEMOCAP dataset.

  • The best performance was obtained with a 2-layer multi-head cross-attention fusion module and both auxiliary tasks, resulting in 78.42% Weighted Accuracy (WA) and 79.71% Unweighted Accuracy (UA).
  • This performance surpasses previous state-of-the-art multimodal models on the same dataset.
  • Even a single auxiliary task (Auxiliary Task 2) showed significant improvement over baseline fusion methods.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the IEMOCAP dataset, while widely used, is limited in size. They note that increasing the complexity of the downstream network (e.g., more cross-attention layers) can lead to performance degradation due to overfitting on this limited data.
  • Generalizability: While the results on IEMOCAP are strong, the generalizability of the proposed method to other datasets or more diverse emotion expressions is not explicitly discussed.
  • Future Work: The paper implicitly suggests exploring larger datasets and potentially more sophisticated auxiliary tasks to further enhance multimodal fusion and emotion recognition capabilities. Investigating the impact of different pre-trained models or fusion architectures could also be a direction for future research.

Summary updated

Abstract

The lack of data and the difficulty of multimodal fusion have always been challenges for multimodal emotion recognition (MER). In this paper, we propose to use pre-trained models as upstream network, wav2vec 2.0 for audio modality and BERT for text modality, and finetune them in downstream task of MER to cope with the lack of data. For the difficulty of multimodal fusion, we use a K-layer multi-head attention mechanism as a downstream fusion module. Starting from the MER task itself, we design two auxiliary tasks to alleviate the insufficient fusion between modalities and guide the network to capture and align emotion-related features. Compared to the previous state-of-the-art models, we achieve a better performance by 78.42% Weighted Accuracy (WA) and 79.71% Unweighted Accuracy (UA) on the IEMOCAP dataset.
BibTeX
@inproceedings{sun2023using,
  author = {Dekai Sun and Yancheng He and Jiqing Han},
  title = {{Using Auxiliary Tasks In Multimodal Fusion of Wav2vec 2.0 And Bert for Multimodal Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=vfp5Tfbitl},
}
Figure