Quantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis

Dimitris Gkoumas, Qiuchi Li, Shahram Dehdashti, Massimo Melucci, Yijun Yu, Dawei Song

Abstract

Multimodal sentiment analysis requires the integration of disparate signals from linguistic, visual, and acoustic sources. Traditional "classical" probability models often fail to capture the nuanced cognitive biases and contextual interferences present in human perception. This paper introduces a decision-level fusion strategy inspired by quantum cognition, treating modalities as incompatible observables. By representing sentiment states in a Hilbert space and utilizing Positive-Operator Valued Measures (POVMs), the model effectively captures "order effects" and interference between modalities. Experimental results on the CMU-MOSI and CMU-MOSEI datasets demonstrate that this quantum-inspired approach significantly outperforms state-of-the-art classical fusion methods, especially in cases of conflicting unimodal signals.

Introduction

Video sentiment analysis is a complex task involving the fusion of text, video, and audio modalities. Classical probability assumes these sources are independent or can be measured simultaneously without mutual interference. However, human cognition often exhibits "order effects"—where the interpretation of one signal (e.g., a facial expression) fundamentally alters the perception of another (e.g., spoken words).

Multimodal data samples consisting of visual frames and acoustic waveforms.

These perspectives are considered "incompatible" in quantum terms. This paper argues that the rigid logic of classical probability is insufficient for modeling such "irrational" human-like decision processes, leading to errors when modalities appear contradictory.

Methodology: Quantum Cognition for Fusion

The researchers adopt the mathematical formalism of Quantum Mechanics (QM) to model uncertainty and interference in decision-making. This approach does not assume the brain is a quantum computer but utilizes QM's superior framework for handling non-classical probability.

Sentiment Hilbert Space

An utterance is represented as a state vector $|S\rangle$ in a two-dimensional complex-valued Hilbert space, where dimensions correspond to "positive" and "negative" sentiment. Before a final judgment, the state exists in a quantum superposition.

Representation of sentiment states as vectors in a Hilbert space.

Incompatible Observables and POVMs

Unimodal classifiers (Text, Video, Audio) are formulated as incompatible observables. Measuring one modality provides a context that physically shifts the system's state before subsequent measurements. To approximate the joint sentiment of all three modalities while accounting for inherent noise, the model employs Positive-Operator Valued Measures (POVMs). This decision-level fusion aggregates unimodal predictions using quantum probability rules rather than simple feature concatenation.

Experimental Evaluation

The model was evaluated on two benchmark datasets:

  • CMU-MOSI: Movie reviews with varying sentiment intensities.
  • CMU-MOSEI: A large-scale dataset with over 23,000 utterances across diverse topics.

Results and Discussion

The proposed quantum-inspired model achieved a new state-of-the-art (SOTA) for decision-level fusion. On the CMU-MOSI dataset, it reached 84.6% accuracy, outperforming the best classical baseline (Stacking at 78.4%). It also surpassed complex content-level models like MulT and RAVEN.

A key insight is the model's ability to handle "extreme cases." In a subset where all three unimodal classifiers failed individually, the quantum model successfully fused the conflicting signals to produce the correct prediction in 31 out of 33 instances. This suggests that quantum interference is a powerful mechanism for resolving sarcasm and other contextual nuances where classical models struggle.

Conclusion and Future Work

While highly effective for binary sentiment, future iterations could expand the Hilbert space dimensions to accommodate multi-class sentiment. Additionally, the authors propose extending this framework to conversational dynamics, where the quantum state of one speaker might influence another, capturing the flow of emotion in dialogue.

Summary updated

Abstract

Video sentiment analysis as a decision-making process is inherently complex, involving the fusion of decisions from multiple modalities and the so-caused cognitive biases. Inspired by recent advances in quantum cognition, we show that the sentiment judgment from one modality could be incompatible with the judgment from another, i.e., the order matters and they cannot be jointly measured to produce a final decision. Thus the cognitive process exhibits ``quantum-like'' biases that cannot be captured by classical probability theories. Accordingly, we propose a fundamentally new, quantum cognitively motivated fusion strategy for predicting sentiment judgments. In particular, we formulate utterances as quantum superposition states of positive and negative sentiment judgments, and uni-modal classifiers as mutually incompatible observables, on a complex-valued Hilbert space with positive-operator valued measures. Experiments on two benchmarking datasets illustrate that our model significantly outperforms various existing decision level and a range of state-of-the-art content-level fusion approaches. The results also show that the concept of incompatibility allows effective handling of all combination patterns, including those extreme cases that are wrongly predicted by all uni-modal classifiers.
BibTeX
@inproceedings{gkoumas2021quantum,
  author = {Dimitris Gkoumas and Qiuchi Li and Shahram Dehdashti and Massimo Melucci and Yijun Yu and Dawei Song},
  title = {{Quantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i1.16165},
}
Figure