Abstract
What does it take to design a machine that learns to answer natural questions about a video? A Video QA system must simultaneously understand language, represent visual content over space-time, and iteratively transform these representations in response to lingual content in the query, finally arriving at a sensible answer. Inspired by the dual-process account of human reasoning, we design a dual process neural architecture composed of a question-guided video processing module (System 1, fast and reactive) followed by a generic reasoning module (System 2, slow and deliberative). System 1 is a hierarchical model that encodes visual patterns about objects, actions, and relations in space-time given textual cues. System 2 performs multi-step inference to iteratively chain visual elements. The system demonstrates competitive results on SVQA and TGIF-QA datasets, particularly in multi-step reasoning tasks.
Introduction: The Challenge of Video QA
Video Question Answering (Video QA) is significantly more complex than its image-based counterpart. It requires a machine to synthesize linguistic understanding with visual content that evolves across both space and time. To answer a question like "What did the man do after he sat down for the second time?", a system must identify objects and actions, track them temporally, and perform multi-step logical operations.
Most current Video QA systems "entangle" these processes within black-box neural networks. This often results in models that over-rely on linguistic biases—guessing answers based on question text alone—or fail when faced with complex, multi-step logic. This research proposes to "disentangle" these stages into a modular architecture inspired by the "dual-process" theory of human cognition.
The Dual-Process Architecture
The proposed model separates fast visual perception from slow, deliberate reasoning through two interconnected systems.
System 1: The "Fast" Perceiver (CRN)
This module handles Clip-based Relational Pattern Recognition. To manage the computational expense and noise of processing every frame, the video is divided into equal-length clips. A question-guided temporal attention mechanism identifies the most relevant frames within each clip. Finally, a "Relation Network" compares these clips at multiple scales (pairs, triplets, etc.), capturing both immediate actions and long-term temporal dependencies.
System 2: The "Slow" Reasoner (MAC)
Once System 1 transforms the video into a structured visual knowledge base, System 2 performs Multi-step Centralized Reasoning. It utilizes Memory-Attention-Composition (MAC) cells—recurrent units that execute a sequence of "read," "control," and "write" operations. In each step, the reasoner focuses on a specific part of the question and queries the visual knowledge base, iteratively refining its understanding to reach a conclusion.
Datasets and Evaluation
The model was rigorously tested on two distinct benchmarks:
- SVQA (Synthetic Video Question Answering): Modeled after the CLEVR benchmark, this dataset features complex questions about geometric objects in motion. It is designed to test pure logic and minimize the impact of language shortcuts.
- TGIF-QA: A large-scale real-world dataset of 165K QA pairs from animated GIFs, covering tasks like repetition counting, state transitions, and frame-level QA.
Key Results
The CRN+MAC model achieved a breakthrough on the SVQA dataset with 75.8% accuracy, nearly doubling the previous state-of-the-art of 44.9%. On TGIF-QA, the model set new records for State Transition (78.7%) and Frame QA (59.2%). Ablation studies confirmed that removing the System 2 reasoning module caused SVQA performance to drop significantly (from 75.8% to 49.3%), highlighting the necessity of the "slow" reasoning component for complex tasks.
Insights and Future Directions
The primary takeaway is that disentanglement is key. By separating perception from reasoning, the model handles compositional logic far more effectively. Modeling relations at the clip level rather than the frame level also proves superior for filtering temporal redundancy and identifying long-term relationships.
However, limitations remain. The current implementation lacks explicit motion features (like optical flow), which affected performance on repetition counting tasks. Future work may involve integrating explicit region proposals for objects and actions, as well as exploring bidirectional feedback loops where the "slow" reasoner can guide the "fast" perceiver's focus more dynamically.
Summary updated