Abstract
Multimodal Sentiment Analysis (MSA) models often achieve high performance on curated benchmark datasets but struggle when faced with real-world "noise." We present Robust-MSA, an interactive visualization platform designed to systematically analyze how imperfections in video, audio, and text modalities degrade model performance. By enabling granular noise injection and real-time feature visualization, Robust-MSA provides researchers with a "debug log" for sentiment AI, highlighting the fragility of state-of-the-art architectures in messy environments.
Introduction
Multimodal Sentiment Analysis (MSA) aims to interpret human emotion by integrating signals from text, audio, and video. While current models demonstrate impressive accuracy on standard benchmarks like CMU-MOSEI, these datasets typically feature "clean" data: centered faces, clear speech, and perfect transcripts.
In real-world applications, data is rarely pristine. Visual occlusions, background noise, and Automatic Speech Recognition (ASR) errors are common. There is currently a significant gap in understanding how these specific imperfections impact model robustness. Robust-MSA addresses this by providing a tool to surgically inject noise and observe the resulting degradation in model predictions and feature extraction.
The Robust-MSA Platform
Robust-MSA is an end-to-end pipeline and interactive dashboard that allows researchers to manipulate multimodal data at the word level. The system supports three primary categories of noise injection:
- Visual Noise: Simulating face detection failures via blank screens or applying Gaussian blur to represent low-quality video.
- Audio Noise: Muting specific segments or adding background noise from the DEMAND dataset (e.g., office, park, or white noise).
- Textual Noise: Simulating ASR errors by replacing or removing specific words.
Beyond noise injection, the platform extracts and plots features in real-time, such as facial Action Units (AUs) and acoustic features (eGeMAPS). This allows users to visualize how noise distorts the underlying data signal before it reaches the model.
Experimental Insights and Results
The platform integrates eight popular MSA models, including MAG-BERT, MISA, and TFR-Net, all pre-trained on the CMU-MOSEI dataset. Through interactive testing, several key insights emerge:
The Fragility of Visual Cues
A critical finding is the extreme sensitivity of facial feature extraction. For instance, simulating a brief face detection failure (using a blank screen) can completely zero out "Action Unit 45" (winking). Because winking is a strong indicator of positive sentiment, its loss can cause an ensemble of models to flip their prediction from "Positive" to "Neutral."
Word-Level Sensitivity
By applying noise to specific words rather than the entire clip, the tool demonstrates that certain parts of a sentence are more "load-bearing" for sentiment than others. This granular analysis helps researchers identify which linguistic or acoustic segments are most vital for correct classification.
Benchmarking and Defense
The system also includes baseline defense mechanisms, such as Fast Fourier Transform (FFT) for audio denoising and motion compensation for video. By comparing "Original," "Modified," and "Defended" features on a single timeline, researchers can evaluate the effectiveness of standard preprocessing techniques in recovering model performance.
Conclusion
Robust-MSA exposes the vulnerability of current state-of-the-art MSA models to real-world noise. By providing a standardized environment for benchmarking eight different architectures against controlled perturbations, the tool serves as a catalyst for developing more robust, noise-aware AI. Future work will focus on expanding the platform to support custom user-uploaded architectures and more sophisticated defense mechanisms.
Summary updated