Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis

Huisheng Mao, Baozheng Zhang, Hua Xu, Ziqi Yuan, Yihe Liu

Abstract

Multimodal Sentiment Analysis (MSA) models often achieve high performance on curated benchmark datasets but struggle when faced with real-world "noise." We present Robust-MSA, an interactive visualization platform designed to systematically analyze how imperfections in video, audio, and text modalities degrade model performance. By enabling granular noise injection and real-time feature visualization, Robust-MSA provides researchers with a "debug log" for sentiment AI, highlighting the fragility of state-of-the-art architectures in messy environments.

Introduction

Multimodal Sentiment Analysis (MSA) aims to interpret human emotion by integrating signals from text, audio, and video. While current models demonstrate impressive accuracy on standard benchmarks like CMU-MOSEI, these datasets typically feature "clean" data: centered faces, clear speech, and perfect transcripts.

In real-world applications, data is rarely pristine. Visual occlusions, background noise, and Automatic Speech Recognition (ASR) errors are common. There is currently a significant gap in understanding how these specific imperfections impact model robustness. Robust-MSA addresses this by providing a tool to surgically inject noise and observe the resulting degradation in model predictions and feature extraction.

The Robust-MSA Platform

Robust-MSA is an end-to-end pipeline and interactive dashboard that allows researchers to manipulate multimodal data at the word level. The system supports three primary categories of noise injection:

  • Visual Noise: Simulating face detection failures via blank screens or applying Gaussian blur to represent low-quality video.
  • Audio Noise: Muting specific segments or adding background noise from the DEMAND dataset (e.g., office, park, or white noise).
  • Textual Noise: Simulating ASR errors by replacing or removing specific words.
The Robust-MSA interface allows for granular noise injection across text, video, and audio modalities.

Beyond noise injection, the platform extracts and plots features in real-time, such as facial Action Units (AUs) and acoustic features (eGeMAPS). This allows users to visualize how noise distorts the underlying data signal before it reaches the model.

Experimental Insights and Results

The platform integrates eight popular MSA models, including MAG-BERT, MISA, and TFR-Net, all pre-trained on the CMU-MOSEI dataset. Through interactive testing, several key insights emerge:

The Fragility of Visual Cues

A critical finding is the extreme sensitivity of facial feature extraction. For instance, simulating a brief face detection failure (using a blank screen) can completely zero out "Action Unit 45" (winking). Because winking is a strong indicator of positive sentiment, its loss can cause an ensemble of models to flip their prediction from "Positive" to "Neutral."

Visualization of how noise-induced feature distortion (e.g., in wink detection) leads to a shift in MSA prediction results.

Word-Level Sensitivity

By applying noise to specific words rather than the entire clip, the tool demonstrates that certain parts of a sentence are more "load-bearing" for sentiment than others. This granular analysis helps researchers identify which linguistic or acoustic segments are most vital for correct classification.

Benchmarking and Defense

The system also includes baseline defense mechanisms, such as Fast Fourier Transform (FFT) for audio denoising and motion compensation for video. By comparing "Original," "Modified," and "Defended" features on a single timeline, researchers can evaluate the effectiveness of standard preprocessing techniques in recovering model performance.

Conclusion

Robust-MSA exposes the vulnerability of current state-of-the-art MSA models to real-world noise. By providing a standardized environment for benchmarking eight different architectures against controlled perturbations, the tool serves as a catalyst for developing more robust, noise-aware AI. Future work will focus on expanding the platform to support custom user-uploaded architectures and more sophisticated defense mechanisms.

Summary updated

Abstract

Improving model robustness against potential modality noise, as an essential step for adapting multimodal models to real-world applications, has received increasing attention among researchers. For Multimodal Sentiment Analysis (MSA), there is also a debate on whether multimodal models are more effective against noisy features than unimodal ones. Stressing on intuitive illustration and in-depth analysis of these concerns, we present Robust-MSA, an interactive platform that visualizes the impact of modality noise as well as simple defence methods to help researchers know better about how their models perform with imperfect real-world data.
BibTeX
@inproceedings{mao2023robust,
  author = {Huisheng Mao and Baozheng Zhang and Hua Xu and Ziqi Yuan and Yihe Liu},
  title = {{Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27078},
}
Figure