Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis

Shiqin Han, Manning Gao, Menghua Jiang, Yuncheng Jiang, Haifeng Hu, Sijie MaiarXiv 2025 · DOI · arXiv

Abstract

Multimodal Sentiment Analysis (MSA) typically involves a trade-off between the high accuracy of Multimodal Large Language Models (MLLMs) and the computational efficiency of smaller specialized models. This paper introduces the Uncertainty-Aware Collaborative System (U-ACS), a hierarchical framework that dynamically routes samples based on predictive uncertainty. By utilizing a lightweight baseline model for "easy" samples and reserving MLLMs for "hard" cases, U-ACS achieves a 3x speedup over standalone MLLMs while improving binary sentiment accuracy. The system employs a novel regression-to-classification entropy measure to guide routing and a collaborative refinement stage for resolving model disagreements.

Introduction

Multimodal Sentiment Analysis (MSA) aims to decipher human emotions by integrating information from text, audio, and visual modalities. As the field evolves, researchers face a significant dilemma: the choice between small, specialized models and massive Multimodal Large Language Models (MLLMs).

Small models are characterized by high inference speeds and low memory footprints, making them ideal for real-time applications. However, they often struggle with noisy data or nuanced emotional expressions. Conversely, MLLMs offer superior reasoning capabilities and generalization but are prohibitively expensive in terms of latency and energy consumption.

Comparison of Small Models vs. MLLMs across various performance and efficiency metrics.

The challenge lies in dynamic resource allocation—identifying which samples require the heavy lifting of an MLLM and which can be handled by a lightweight model. The Uncertainty-Aware Collaborative System (U-ACS) addresses this by implementing an uncertainty-guided routing mechanism.

Methodology: The U-ACS Framework

The U-ACS architecture is a three-stage hierarchical pipeline designed to optimize the balance between performance and cost.

The U-ACS pipeline, illustrating the flow from UBM processing to MLLM verification and collaborative refinement.

1. Uncertainty-aware Baseline Model (UBM)

The first stage involves a lightweight UBM that utilizes BERT for linguistic features and LSTMs for audio-visual data. Because MSA is traditionally a regression task, calculating standard uncertainty metrics like Shannon Entropy is non-trivial. The authors innovate by adding an auxiliary classification head that discretizes sentiment into three categories (Positive, Negative, Neutral). This allows the system to generate a principled uncertainty score ($u_s$).

2. Hierarchical Routing and Verification

  • Stage 1 (Fast Path): If the UBM's uncertainty score is below a predefined threshold ($\tau_1$), the prediction is accepted immediately, bypassing the MLLM entirely.
  • Stage 2 (MLLM Processing): Samples with high uncertainty are escalated to an MLLM (such as HumanOmni or VideoLLaMA2). If the MLLM is confident ($u_l \le \tau_2$) or if its sentiment polarity aligns with the UBM, the result is finalized.

3. Collaborative Refinement

In cases where the models disagree or remain uncertain, a third stage is triggered. This "Collaborative Processing" involves a second MLLM pass using an enhanced prompt. This prompt includes the previous predictions and uncertainty scores, tasking the MLLM to act as a "referee" to resolve the conflict and provide a stable final label.

Experimental Evaluation

The system was rigorously tested on three benchmark datasets:

  • CMU-MOSI: A dataset of 2,199 video segments with fine-grained sentiment labels.
  • CMU-MOSEI: A larger-scale dataset containing over 23,000 segments from 1,000 speakers.
  • CH-SIMS: A Chinese multimodal dataset used to evaluate cross-cultural and cross-linguistic robustness.

Key Results and Insights

The U-ACS framework demonstrated significant improvements in both efficiency and accuracy:

  • Efficiency: On the MOSI dataset, U-ACS achieved a 3x speedup, reducing inference time to 33% of a standalone MLLM.
  • Accuracy: Surprisingly, the collaborative approach outperformed standalone MLLMs in binary accuracy (Acc2). On MOSI, U-ACS reached 91.8% Acc2, compared to 91.3% for the standalone HumanOmni model.
  • Robustness: The system proved model-agnostic, showing consistent gains across different MLLM backbones like HumanOmni and VideoLLaMA2.

A critical insight from the study is the effectiveness of "Polarity Agreement." The authors found that if a small model and a large model agree on the direction of sentiment, the prediction is highly reliable, allowing for early exits from the pipeline without sacrificing quality.

Conclusion and Future Work

U-ACS provides a pragmatic path forward for deploying advanced multimodal systems in resource-constrained environments. By intelligently routing samples, it captures the best of both worlds: the speed of small models and the intelligence of MLLMs.

Future research will focus on refining the threshold calibration process ($\tau_1, \tau_2$) and exploring the application of this collaborative framework to other complex multimodal tasks, such as fine-grained emotion recognition and intent detection.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have notably enhanced the performance of Multimodal Sentiment Analysis (MSA), yet their massive parameter scale leads to excessive resource consumption in training and inference, severely limiting model efficiency. To balance performance and efficiency for MSA, this paper innovatively proposes a novel Uncertainty-Aware Collaborative System (U-ACS) that integrates Uncertainty-aware Baseline Model (UBM) with MLLMs. U-ACS operates in three stages: First, all samples are processed by the UBM, retain high-confidence samples and forward low-confidence samples to the MLLM. Notably, to address the challenge that continuous outputs of regression tasks hinder uncertainty calculation, we innovatively convert the continuous sentiment label prediction task to a classification task, enabling a more accurate calculation of entropy and uncertainty. Second, the MLLM performs initial process. In this stage, high-confidence samples or low-confidence samples whose predictive sentiment polarity matches that of the UBM are deemed acceptable, while unqualified samples are forwarded for further processing. Finally, the MLLM performs secondary inference on remaining low-confidence samples using prompts augmented with prior rounds predictions as references. By aggregating results from the three stages, U-ACS preserves high MSA prediction accuracy while drastically boosting efficiency via offloading most simple samples to the UBM and minimizing MLLM processing volume. Extensive experiments verify that U-ACS maintains superior performance while significantly reducing computational overhead and resource consumption.
BibTeX
@misc{han2025uncertainty,
  author = {Shiqin Han and Manning Gao and Menghua Jiang and Yuncheng Jiang and Haifeng Hu and Sijie Mai},
  title = {{Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis}},
  year = {2025},
  eprint = {2509.04459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.04459},
}
Figure