Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning

Songning Lai, Xifeng Hu, Yulong Li, Zhaoxia Ren, Zhi Liu, Danmin Miao

TL;DR

This paper introduces a novel approach for multimodal sentiment analysis that effectively disentangles shared and modality-specific (private) information. By employing a deep modal shared information learning module using covariance matrices and a self-supervised strategy for private information, the model achieves state-of-the-art performance on benchmark datasets, offering a more nuanced understanding of emotions across text, audio, and visual data.

What Problem Does This Paper Address?

Multimodal sentiment analysis (MSA) aims to understand human emotions by combining information from various sources like text, audio, and video. A key challenge in MSA is effectively learning representations that capture both the information common across all modalities (shared information) and the unique information present in each individual modality (private information). Traditional methods often struggle with this disentanglement, especially when dealing with raw feature fusion and uniform labels. This can lead to a loss of modality-specific nuances, hindering accurate sentiment prediction. The paper addresses this by proposing a method to explicitly learn and leverage both shared and private information for improved MSA.

Methods

The authors propose a novel framework for multimodal sentiment analysis with two core components:

  • Deep Modal Shared Information Learning Module: This module uses the covariance matrix as a second-order statistic to measure the distribution of features between aligned modalities. By analyzing the covariance matrices of different modalities (e.g., audio and visual), the module quantifies and learns the shared information content between them. A differentiable loss function based on the Frobenius norm of the difference between covariance matrices is introduced to facilitate this learning.
  • Self-supervised Unimodal Label Generation Module (ULGM): This module, inspired by previous work, generates unimodal labels using a self-supervised learning strategy. This helps the model focus on capturing private information specific to each modality, complementing the shared information learned by the other module.

These modules are integrated into a multi-task learning framework. The overall objective function combines losses for the multimodal task, unimodal tasks (guided by ULGM), and the alignment of deep modal features.

Datasets

The proposed model was evaluated on three common multimodal sentiment analysis benchmark datasets:

  • CMU-MOSI: A widely used dataset for multimodal sentiment analysis, featuring YouTube videos with text, audio, and visual data.
  • CMU-MOSEI: A larger and more comprehensive dataset than CMU-MOSI, also containing multimodal data from YouTube videos, offering a wider range of topics and emotional expressions.
  • SIMS: A recently introduced Chinese multimodal sentiment analysis dataset, notable for its unaligned data and fine-grained annotations.

Key Insights and Ideas

  • Covariance Matrix for Shared Information: Using the covariance matrix is an elegant way to capture statistical dependencies and shared information between modalities, going beyond simple feature concatenation.
  • Disentangling Shared and Private Information is Crucial: The paper strongly suggests that explicitly learning both shared and private information leads to better performance than methods that only focus on one or rely on raw fusion.
  • Self-supervised Label Generation for Private Information: The ULGM module effectively guides the model to learn modality-specific nuances, which are vital for a comprehensive understanding of sentiment.
  • Multi-task Learning Enhances Differentiation: The multi-task learning strategy helps the model focus on distinguishing between modalities, further refining the learning of both shared and private features.
  • Flexibility in Information Exchange: The proposed module allows for adjustable parameter control, enabling the model to prioritize learning either shared or private information as needed for specific tasks or modalities.

Key Results

The experimental results on the CMU-MOSI, CMU-MOSEI, and SIMS datasets demonstrate the effectiveness of the proposed approach. The paper reports that their model achieves state-of-the-art performance, outperforming several existing advanced multimodal sentiment analysis models across various metrics, including MAE, Correlation, Accuracy (Acc-2), and F1-Score. For instance, on the SIMS dataset, their model showed superior performance compared to TFN, LMF, and Self-MM. The results also highlight the importance of data alignment, showing that aligned data simplifies the task, but their model performs competitively even on unaligned data.

Limitations and Future Work

The authors acknowledge a limitation in their reliance on uniform multimodal labels and raw feature fusion. They suggest that future research could explore alternative fusion techniques and investigate different methods for label generation. While the model demonstrates strong performance, assessing its generalizability on diverse datasets is highlighted as a crucial area for future work. The paper also suggests extending the application of their approach to other multimodal tasks beyond sentiment analysis.

Summary updated

Abstract

Designing an effective representation learning method for multimodal sentiment analysis tasks is a crucial research direction. The challenge lies in learning both shared and private information in a complete modal representation, which is difficult with uniform multimodal labels and a raw feature fusion approach. In this work, we propose a deep modal shared information learning module based on the covariance matrix to capture the shared information between modalities. Additionally, we use a label generation module based on a self-supervised learning strategy to capture the private information of the modalities. Our module is plug-and-play in multimodal tasks, and by changing the parameterization, it can adjust the information exchange relationship between the modes and learn the private or shared information between the specified modes. We also employ a multi-task learning strategy to help the model focus its attention on the modal differentiation training data. We provide a detailed formulation derivation and feasibility proof for the design of the deep modal shared information learning module. We conduct extensive experiments on three common multimodal sentiment analysis baseline datasets, and the experimental results validate the reliability of our model. Furthermore, we explore more combinatorial techniques for the use of the module. Our approach outperforms current state-of-the-art methods on most of the metrics of the three public datasets.
BibTeX
@inproceedings{lai2023shared,
  author = {Songning Lai and Xifeng Hu and Yulong Li and Zhaoxia Ren and Zhi Liu and Danmin Miao},
  title = {{Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning}},
  booktitle = {IJCNN},
  year = {2023},
  url = {https://openreview.net/forum?id=MgsQMhI12l5},
}
Figure