QENN: A Quantum Entanglement-Inspired Neural Network for Interaction and Relationship Prediction in Story Videos

Zijun Xu, Zhengqian Wu, Chunjie Zhang, Zhongyuan Wang, Chunxia Xiao, Chao LiangCVPR 2026 Findings 2026 · OpenReview

TL;DR

The Quantum Entanglement-Inspired Neural Network (QENN) introduces a novel framework for predicting character interactions and relationships in story videos. By leveraging quantum computing principles, QENN models the inherently inseparable correlations between characters and across modalities (video and dialogue), consistently outperforming state-of-the-art 7B multimodal large language models and traditional feature fusion techniques.

What Problem Does This Paper Address?

Understanding how characters interact and relate to one another in a movie or TV show is a complex multimodal task. It requires analyzing subtle visual cues alongside spoken dialogue.

Existing video understanding methods typically fall into two traps:

  1. The Holistic Approach: They analyze the entire video frame at once, which introduces significant background noise and obscures subtle social cues.
  2. The Separable Approach: They extract individual character and modality features independently and simply concatenate them.

The authors argue that the separable approach fails because social interactions are fundamentally inseparable. For example, as shown in Figure 1, a character "ignoring" someone cannot be deduced by looking at either character's actions or the dialogue in isolation; it only becomes clear when the behaviors are modeled as a tightly coupled, unified system.

The "ignores" interaction in the movie Liberty Kid, demonstrating the inseparable nature of character dynamics.

Methods

QENN addresses this by mimicking the three core stages of the quantum computing paradigm: State Preparation, State Entanglement, and State Measurement.

Model architecture of QENN showing the state preparation, entanglement, and measurement pipeline.
  1. State Preparation: Classical features are extracted from the video (using BERT for text and Temporal Shift Networks for video/character tracks). These real-valued features are projected into a complex vector space ($\mathbb{C}^{N_d}$) using dedicated Multi-Layer Perceptrons (MLPs) and $L_2$-normalized to represent valid quantum states.
  2. State Entanglement: To model inseparable correlations, the system creates composite states using tensor products ($\otimes$) for character-character pairs and vision-text pairs. It then applies learnable unitary transformation layers ($U_{cc}$ and $U_{vt}$) to entangle these states. This ensures the features can no longer be mathematically factored back into isolated, independent components. The foreground (characters) and background (vision-text) entangled states are then combined via a direct sum ($\oplus$).
  3. State Measurement: The augmented entangled state is mapped into interaction and relationship subspaces. QENN then performs projective measurements onto learnable orthonormal bases. This simulates the quantum "collapse" of a superposition state into a definitive classical probability distribution for final classification.

Optimization Note: Because standard backpropagation violates the strict mathematical constraints of unitary matrices and orthonormal bases, the authors utilize Riemannian optimization on the complex Stiefel manifold to update the network weights.

Datasets

The model was evaluated on four public Deep Video Understanding (DVU) benchmarks, which contain a total of 24 unique movies and over 3,000 interaction/relationship questions:

  • ACM Multimedia DVU Grand Challenge 2021
  • ACM Multimedia DVU Grand Challenge 2022
  • NIST TRECVID DVU Task 2022
  • NIST TRECVID DVU Task 2023

Evaluation metrics included Accuracy (ACC) for single-answer questions and Mean Reciprocal Rank (MRR) for option-ranking questions.

Key Insights and Ideas

  • Inseparability via Quantum Mechanics: The core "aha moment" is that quantum entanglement mathematically prevents a composite state from being decomposed into its constituent subsystems. This perfectly mirrors cognitive psychology theories regarding human relationship perception.
  • Foreground vs. Background Context: Treating the character-character pair as the "foreground" and the vision-text modality pair as the "background" context allows the network to focus on the subjects while still absorbing environmental and conversational cues.
  • Manifold Optimization is Key: Standard gradient descent breaks the properties required for quantum simulation (like unitarity). Shifting to Riemannian optimization on the Stiefel manifold allows deep learning architectures to successfully preserve these quantum constraints during training.

Key Results

QENN achieved a new state-of-the-art across all four DVU datasets, demonstrating a clear advantage over both traditional fusion methods and massive pre-trained models:

  • Vs. Multimodal Large Language Models (MLLMs): On the TRECVID 23 dataset, QENN achieved an average score of 41.56%, vastly outperforming the strongest 7B MLLM baseline, ViLAMP, which scored 23.70%.
  • Vs. Classical Fusion: QENN consistently beat seven traditional fusion schemes (including Bilinear Pooling, Self-Attention, and simple concatenation) across all datasets, proving that quantum-inspired entanglement captures higher-order interplays better than standard architectures.
  • Vs. Competition Champions: QENN outperformed the official "Champion Teams" on three out of four datasets, notably reversing the trend on the Grand Challenge 2022 relationship task (57.50% vs. 39.20%) without relying on massive external pre-training datasets like HowTo100M.

Limitations and Future Work

The authors note two primary limitations:

  1. Pairwise Restriction: The current formulation only models interactions between exactly two characters, leaving multi-party social dynamics unexplored.
  2. Static Relationships: The model treats relationships as static attributes across video segments rather than tracking how relationships dynamically evolve over the course of a full-length movie narrative.

Future work will focus on extending the quantum framework to multi-party scenarios and introducing temporal modeling to capture relationship evolution.

Summary updated

Abstract

Learning character interactions and relationships in story videos is challenging, as it requires a joint understanding of both the complex character dynamics and the complementary information from different modalities, such as visual scenes and textual dialogues. Existing methods typically model these aspects by analyzing the entire video content, which is prone to noise, or by processing different data sources independently, which fails to capture their inherent connections. This separability stands in contrast to the inseparable nature of these elements, as revealed by studies in cognitive psychology. Inspired by quantum entanglement theory for modeling such inseparable relations, this paper proposes a Quantum Entanglement-inspired Neural Network (QENN) to uniformly model: (1) character correlation within interactions and relationships in stories; (2) modality correlation between visual content and textual dialogue in videos. Specifically, QENN mimics the general quantum computing paradigm by first entangling the separable feature via unitary transformations, and then applying orthogonal projections to ``collapse" these entangled states into discriminative representations for prediction. Extensive evaluations on four public datasets demonstrate that QENN consistently outperforms existing baseline methods.
BibTeX
@inproceedings{xu2026qenn,
  author = {Zijun Xu and Zhengqian Wu and Chunjie Zhang and Zhongyuan Wang and Chunxia Xiao and Chao Liang},
  title = {{QENN: A Quantum Entanglement-Inspired Neural Network for Interaction and Relationship Prediction in Story Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=T1kCtAvtmC},
}
Figure