Abstract
Designing machines that can answer natural language questions about video requires a system to simultaneously understand language, represent visual content over space-time, and iteratively transform these representations. Inspired by the dual-process account of human reasoning, we propose a neural architecture composed of a question-guided video processing module (System 1, fast and reactive) followed by a generic reasoning module (System 2, slow and deliberative). System 1 uses a hierarchical Clip-based Relational Network (CRN) to encode visual patterns about objects, actions, and relations. System 2 uses a Memory-Attention-Composition (MAC) network to perform multi-step inference, chaining visual elements as instructed by the query. Our system demonstrates competitive results on the SVQA and TGIF-QA datasets, particularly in complex multi-step reasoning tasks.
Introduction
Answering natural language questions about video is significantly more challenging than static image QA. A successful system must track objects through space-time, recognize actions, and understand temporal relationships (e.g., "What happened after the man sat down?"). Most existing Video QA models "entangle" pattern recognition and reasoning, often relying on black-box recurrent neural networks (RNNs). These systems frequently fall prey to "linguistic bias," guessing answers based on question wording rather than visual evidence.
The Dual-Process Architecture
To address these challenges, we separate visual perception from logical reasoning, mimicking the dual-process model of human cognition.
System 1: Clip-based Relational Network (CRN)
Instead of treating a video as a flat sequence of frames, the CRN builds a hierarchy. It divides the video into clips and uses temporal attention to extract relevant features guided by the question. A relational network then models $k$-order relations between these clips, capturing both short-term actions and long-term temporal dependencies efficiently.
System 2: MAC Network
The output of the CRN serves as a "knowledge base" for the Memory-Attention-Composition (MAC) network. As a generic reasoning engine, the MAC network iteratively performs "Read," "Control," and "Write" operations to chain together visual elements. This allows for complex, compositional reasoning, such as identifying an object based on its relationship to another previously identified object.
Experimental Setup and Datasets
We evaluated the model on two distinct benchmarks:
- SVQA (Synthetic Video Question Answering): Designed to test multi-step reasoning with minimal linguistic bias, featuring complex logical structures.
- TGIF-QA: A large-scale real-world dataset containing 165K QA pairs from 72K animated GIFs, testing repetition counting, action recognition, and state transitions.
Results and Analysis
The dual-process approach yielded significant performance gains:
- SVQA: The CRN+MAC model achieved 75.8% overall accuracy, a massive jump from the previous best baseline of 44.9%.
- TGIF-QA: The model set new state-of-the-art records for State Transition (78.7%) and Frame QA (59.2%).
Ablation studies confirmed that both systems are essential. Removing the multi-step reasoning (MAC) or the hierarchical video modeling (CRN) significantly degraded performance, proving that disentangling vision from logic is key to handling complex Video QA tasks.
Conclusion and Future Work
By separating the "vision" part (CRN) from the "thinking" part (MAC), our model becomes more modular and resilient to linguistic bias. While the model excels at logical reasoning using appearance features, future work could incorporate explicit motion features (like C3D) to improve action counting. Additionally, exploring "active" perception—where System 2 can direct System 1 to re-examine specific video segments—could further resolve visual ambiguities.
Summary updated