Abstract
Video sentiment analysis as a decision-making process is inherently complex, involving the fusion of decisions from multiple modalities and the so-caused cognitive biases. Inspired by recent advances in quantum cognition, we show that the sentiment judgment from one modality could be incompatible with the judgment from another, i.e., the order matters and they cannot be jointly measured to produce a final decision. Thus the cognitive process exhibits "quantum-like" biases that cannot be captured by classical probability theories. Accordingly, we propose a fundamentally new, quantum cognitively motivated fusion strategy for predicting sentiment judgments. In particular, we formulate utterances as quantum superposition states of positive and negative sentiment judgments, and uni-modal classifiers as mutually incompatible observables, on a complex-valued Hilbert space with positive-operator valued measures. Experiments on two benchmarking datasets illustrate that our model significantly outperforms various existing decision level and a range of state-of-the-art content-level fusion approaches. The results also show that the concept of incompatibility allows effective handling of all combination patterns, including those extreme cases that are wrongly predicted by all uni-modal classifiers.
Introduction
In multimodal sentiment analysis, the goal is to determine the sentiment of a video clip by integrating information from text, audio, and visual channels. Traditional AI approaches typically treat this as a straightforward mathematical fusion problem, assuming that different modalities are independent and can be measured simultaneously without interference. However, human judgment is rarely so linear; it is contextual and prone to cognitive biases.
This research introduces a novel framework based on Quantum Cognition to model the "irrational" ways humans process information. By moving beyond classical probability theories, the proposed model captures the interference between different modalities—where the perception of one (e.g., what is said) fundamentally changes the interpretation of another (e.g., how it is said).
Methodology: Modeling Incompatibility
The core innovation of this work is the treatment of linguistic ($L$), visual ($V$), and acoustic ($A$) modalities as incompatible observables. In quantum terms, incompatibility means that the order of measurement matters, and the act of judging one modality influences the state of the others.
As shown in the figure above, the model represents an utterance as a pure state ($|S\rangle$) in a 2-dimensional complex-valued Hilbert space. This state exists in a superposition of positive and negative sentiment until it is measured. Because the modalities represent different, non-orthogonal coordinate systems, they cannot be jointly measured in a classical sense. The researchers utilize Positive-Operator Valued Measures (POVMs) to approximate the sentiment of all three modalities simultaneously while accounting for the "noise" or uncertainty caused by their interaction. This allows the model to learn the geometric relationships (angles) between modalities, mapping how much they interfere with one another.
Experimental Results and Discussion
The model was evaluated on two gold-standard benchmarks: CMU-MOSI (2,199 utterances) and CMU-MOSEI (23,453 utterances). The results demonstrate a significant performance leap over traditional decision-level and content-level fusion techniques.
As detailed in the table, the proposed model reached 84.6% accuracy on CMU-MOSI and 84.9% accuracy on CMU-MOSEI. This represents a substantial improvement over classical ensemble methods like Stacking (78.4% on MOSI) and Deep Fusion (77.8% on MOSI).
One of the most striking findings is the model's robustness in "extreme cases." In instances where all three individual uni-modal classifiers (text, audio, and visual) failed to predict the correct sentiment, the quantum fusion model was often able to find the correct signal. This is attributed to the model's ability to capture "interference"—finding positive or negative sentiment in the complex interaction between modalities that was invisible to individual sensors.
Conclusion
By applying the mathematical formalism of quantum mechanics to cognitive processes, this research provides a more flexible framework for modeling human-like sentiment judgment. The concept of incompatibility allows the model to handle complex combination patterns that classical probability fails to capture. Future work aims to extend this framework to multi-class emotion recognition and conversational settings, where the interactions between different speakers add further layers of complexity.
Summary updated