A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis

Chuanqi Tao, Jiaming Li, Tianzi Zang, Peng GaoAAAI 2025 · DOI

Abstract

Multimodal Sentiment Analysis (MSA) aims to understand human emotion by integrating information from text, audio, and video modalities. However, real-world data is often incomplete due to sensor malfunctions or environmental noise, causing standard models to fail. We introduce the Multi-Focus-Driven Multi-Branch Network (MFMB-Net), a robust framework designed to handle missing data through a collaborative multi-scale approach. MFMB-Net features a three-branch architecture that simultaneously fuses global (macro) and local (micro) features while a dedicated reconstruction branch restores missing information. Our experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that MFMB-Net achieves state-of-the-art performance and exhibits significant resilience to missing modalities.

Introduction

Multimodal Sentiment Analysis is essential for detecting nuanced human emotions, such as sarcasm or joy, by combining linguistic, acoustic, and visual cues. While existing models perform well with "perfect" data, they often crumble when a modality is missing or corrupted. Researchers from the Nanjing University of Aeronautics and Astronautics have developed MFMB-Net to address this vulnerability. Unlike previous methods that treat data reconstruction and fusion as separate steps, MFMB-Net integrates them into a unified process that captures both the "big picture" global context and "fine-grained" local details.

Methodology: The Multi-Branch Architecture

The core innovation of MFMB-Net lies in its three-stream processing pipeline, which allows the model to maintain high-fidelity representations even when input data is sparse.

The MFMB-Net architecture, featuring Macro-Fusion, Micro-Fusion, and Reconstruction branches.
  1. Macro-Fusion Stream: This branch focuses on the global dimension of the input. It utilizes a Transformer-based "coarse hub" to aggregate high-level information across modalities, compressing high-dimensional data into a compact latent space where critical cross-modal exchanges occur.
  2. Micro-Fusion Stream: To capture subtle, short-term emotional cues, this stream targets local features. It employs a pseudo-Siamese network with a Memory-Gating Network (MGN) to filter noise and a "Coarse-to-fine" MLP module to refine temporal details.
  3. Reconstruction Module: This branch uses a key-value memory network to restore missing features. By interacting directly with the fusion streams, it ensures that reconstructed data remains consistent with the available modalities, effectively "filling in the blanks" during inference.

The model is optimized using a dual-loss objective: a Task Loss for sentiment prediction and a Generative Loss (Smooth L1) to ensure the accuracy of the reconstruction process.

Experimental Setup and Results

The performance of MFMB-Net was evaluated on two standard benchmarks: CMU-MOSI (2,199 monologue clips) and CMU-MOSEI (22,856 segments). To simulate real-world conditions, the researchers applied random masking to the input modalities at rates ranging from 0% to 100%.

Comparison of MFMB-Net against state-of-the-art models on CMU-MOSI and CMU-MOSEI datasets.

As shown in the results table, MFMB-Net consistently outperforms state-of-the-art models like MMIM and TFR-Net across nearly all metrics, including Mean Absolute Error (MAE), Correlation (Corr), and F1 Score. Notably, on the CMU-MOSEI dataset, MFMB-Net improved the F1 score by 2.2% over previous benchmarks under missing modality conditions. Ablation studies further confirmed the importance of the reconstruction module, which, when removed, led to a 1.4% drop in seven-class accuracy.

Discussion and Conclusion

MFMB-Net demonstrates that the synergy between global context and local detail is vital for robust sentiment analysis. By leveraging text as a high-density "anchor" modality, the model effectively filters redundant information in audio and video streams.

While the model shows significant improvements, challenges remain regarding data imbalance in binary classification and the potential for incorporating external knowledge bases to further enhance contextual understanding. Future work will focus on optimizing the multi-branch structure for computational efficiency, facilitating deployment on edge devices where sensor failures are most common.

Summary updated

BibTeX
@inproceedings{tao2025multi,
  author = {Chuanqi Tao and Jiaming Li and Tianzi Zang and Peng Gao},
  title = {{A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32146},
}
Figure