Abstract
Multimodal Sentiment Analysis (MSA) typically involves a trade-off between the high accuracy of Multimodal Large Language Models (MLLMs) and the computational efficiency of smaller specialized models. This paper introduces the Uncertainty-Aware Collaborative System (U-ACS), a hierarchical framework that dynamically routes samples based on predictive uncertainty. By utilizing a lightweight baseline model for "easy" samples and reserving MLLMs for "hard" cases, U-ACS achieves a 3x speedup over standalone MLLMs while improving binary sentiment accuracy. The system employs a novel regression-to-classification entropy measure to guide routing and a collaborative refinement stage for resolving model disagreements.
Introduction
Multimodal Sentiment Analysis (MSA) aims to decipher human emotions by integrating information from text, audio, and visual modalities. As the field evolves, researchers face a significant dilemma: the choice between small, specialized models and massive Multimodal Large Language Models (MLLMs).
Small models are characterized by high inference speeds and low memory footprints, making them ideal for real-time applications. However, they often struggle with noisy data or nuanced emotional expressions. Conversely, MLLMs offer superior reasoning capabilities and generalization but are prohibitively expensive in terms of latency and energy consumption.
The challenge lies in dynamic resource allocation—identifying which samples require the heavy lifting of an MLLM and which can be handled by a lightweight model. The Uncertainty-Aware Collaborative System (U-ACS) addresses this by implementing an uncertainty-guided routing mechanism.
Methodology: The U-ACS Framework
The U-ACS architecture is a three-stage hierarchical pipeline designed to optimize the balance between performance and cost.
1. Uncertainty-aware Baseline Model (UBM)
The first stage involves a lightweight UBM that utilizes BERT for linguistic features and LSTMs for audio-visual data. Because MSA is traditionally a regression task, calculating standard uncertainty metrics like Shannon Entropy is non-trivial. The authors innovate by adding an auxiliary classification head that discretizes sentiment into three categories (Positive, Negative, Neutral). This allows the system to generate a principled uncertainty score ($u_s$).
2. Hierarchical Routing and Verification
- Stage 1 (Fast Path): If the UBM's uncertainty score is below a predefined threshold ($\tau_1$), the prediction is accepted immediately, bypassing the MLLM entirely.
- Stage 2 (MLLM Processing): Samples with high uncertainty are escalated to an MLLM (such as HumanOmni or VideoLLaMA2). If the MLLM is confident ($u_l \le \tau_2$) or if its sentiment polarity aligns with the UBM, the result is finalized.
3. Collaborative Refinement
In cases where the models disagree or remain uncertain, a third stage is triggered. This "Collaborative Processing" involves a second MLLM pass using an enhanced prompt. This prompt includes the previous predictions and uncertainty scores, tasking the MLLM to act as a "referee" to resolve the conflict and provide a stable final label.
Experimental Evaluation
The system was rigorously tested on three benchmark datasets:
- CMU-MOSI: A dataset of 2,199 video segments with fine-grained sentiment labels.
- CMU-MOSEI: A larger-scale dataset containing over 23,000 segments from 1,000 speakers.
- CH-SIMS: A Chinese multimodal dataset used to evaluate cross-cultural and cross-linguistic robustness.
Key Results and Insights
The U-ACS framework demonstrated significant improvements in both efficiency and accuracy:
- Efficiency: On the MOSI dataset, U-ACS achieved a 3x speedup, reducing inference time to 33% of a standalone MLLM.
- Accuracy: Surprisingly, the collaborative approach outperformed standalone MLLMs in binary accuracy (Acc2). On MOSI, U-ACS reached 91.8% Acc2, compared to 91.3% for the standalone HumanOmni model.
- Robustness: The system proved model-agnostic, showing consistent gains across different MLLM backbones like HumanOmni and VideoLLaMA2.
A critical insight from the study is the effectiveness of "Polarity Agreement." The authors found that if a small model and a large model agree on the direction of sentiment, the prediction is highly reliable, allowing for early exits from the pipeline without sacrificing quality.
Conclusion and Future Work
U-ACS provides a pragmatic path forward for deploying advanced multimodal systems in resource-constrained environments. By intelligently routing samples, it captures the best of both worlds: the speed of small models and the intelligence of MLLMs.
Future research will focus on refining the threshold calibration process ($\tau_1, \tau_2$) and exploring the application of this collaborative framework to other complex multimodal tasks, such as fine-grained emotion recognition and intent detection.
Summary updated