Abstract
Multimodal sentiment analysis requires the integration of disparate signals from linguistic, visual, and acoustic sources. Traditional "classical" probability models often fail to capture the nuanced cognitive biases and contextual interferences present in human perception. This paper introduces a decision-level fusion strategy inspired by quantum cognition, treating modalities as incompatible observables. By representing sentiment states in a Hilbert space and utilizing Positive-Operator Valued Measures (POVMs), the model effectively captures "order effects" and interference between modalities. Experimental results on the CMU-MOSI and CMU-MOSEI datasets demonstrate that this quantum-inspired approach significantly outperforms state-of-the-art classical fusion methods, especially in cases of conflicting unimodal signals.
Introduction
Video sentiment analysis is a complex task involving the fusion of text, video, and audio modalities. Classical probability assumes these sources are independent or can be measured simultaneously without mutual interference. However, human cognition often exhibits "order effects"—where the interpretation of one signal (e.g., a facial expression) fundamentally alters the perception of another (e.g., spoken words).
These perspectives are considered "incompatible" in quantum terms. This paper argues that the rigid logic of classical probability is insufficient for modeling such "irrational" human-like decision processes, leading to errors when modalities appear contradictory.
Methodology: Quantum Cognition for Fusion
The researchers adopt the mathematical formalism of Quantum Mechanics (QM) to model uncertainty and interference in decision-making. This approach does not assume the brain is a quantum computer but utilizes QM's superior framework for handling non-classical probability.
Sentiment Hilbert Space
An utterance is represented as a state vector $|S\rangle$ in a two-dimensional complex-valued Hilbert space, where dimensions correspond to "positive" and "negative" sentiment. Before a final judgment, the state exists in a quantum superposition.
Incompatible Observables and POVMs
Unimodal classifiers (Text, Video, Audio) are formulated as incompatible observables. Measuring one modality provides a context that physically shifts the system's state before subsequent measurements. To approximate the joint sentiment of all three modalities while accounting for inherent noise, the model employs Positive-Operator Valued Measures (POVMs). This decision-level fusion aggregates unimodal predictions using quantum probability rules rather than simple feature concatenation.
Experimental Evaluation
The model was evaluated on two benchmark datasets:
- CMU-MOSI: Movie reviews with varying sentiment intensities.
- CMU-MOSEI: A large-scale dataset with over 23,000 utterances across diverse topics.
Results and Discussion
The proposed quantum-inspired model achieved a new state-of-the-art (SOTA) for decision-level fusion. On the CMU-MOSI dataset, it reached 84.6% accuracy, outperforming the best classical baseline (Stacking at 78.4%). It also surpassed complex content-level models like MulT and RAVEN.
A key insight is the model's ability to handle "extreme cases." In a subset where all three unimodal classifiers failed individually, the quantum model successfully fused the conflicting signals to produce the correct prediction in 31 out of 33 instances. This suggests that quantum interference is a powerful mechanism for resolving sarcasm and other contextual nuances where classical models struggle.
Conclusion and Future Work
While highly effective for binary sentiment, future iterations could expand the Hilbert space dimensions to accommodate multi-class sentiment. Additionally, the authors propose extending this framework to conversational dynamics, where the quantum state of one speaker might influence another, capturing the flow of emotion in dialogue.
Summary updated