Neural Reasoning, Fast and Slow, for Video Question Answering

Thao Minh Le, Vuong Le, Svetha Venkatesh, Truyen Tran

Abstract

What does it take to design a machine that learns to answer natural questions about a video? A Video QA system must simultaneously understand language, represent visual content over space-time, and iteratively transform these representations in response to lingual content in the query, finally arriving at a sensible answer. Inspired by the dual-process account of human reasoning, we design a dual process neural architecture composed of a question-guided video processing module (System 1, fast and reactive) followed by a generic reasoning module (System 2, slow and deliberative). System 1 is a hierarchical model that encodes visual patterns about objects, actions, and relations in space-time given textual cues. System 2 performs multi-step inference to iteratively chain visual elements. The system demonstrates competitive results on SVQA and TGIF-QA datasets, particularly in multi-step reasoning tasks.

Introduction: The Challenge of Video QA

Video Question Answering (Video QA) is significantly more complex than its image-based counterpart. It requires a machine to synthesize linguistic understanding with visual content that evolves across both space and time. To answer a question like "What did the man do after he sat down for the second time?", a system must identify objects and actions, track them temporally, and perform multi-step logical operations.

Most current Video QA systems "entangle" these processes within black-box neural networks. This often results in models that over-rely on linguistic biases—guessing answers based on question text alone—or fail when faced with complex, multi-step logic. This research proposes to "disentangle" these stages into a modular architecture inspired by the "dual-process" theory of human cognition.

The Dual-Process Architecture

The proposed model separates fast visual perception from slow, deliberate reasoning through two interconnected systems.

System 1: The "Fast" Perceiver (CRN)

This module handles Clip-based Relational Pattern Recognition. To manage the computational expense and noise of processing every frame, the video is divided into equal-length clips. A question-guided temporal attention mechanism identifies the most relevant frames within each clip. Finally, a "Relation Network" compares these clips at multiple scales (pairs, triplets, etc.), capturing both immediate actions and long-term temporal dependencies.

System 2: The "Slow" Reasoner (MAC)

Once System 1 transforms the video into a structured visual knowledge base, System 2 performs Multi-step Centralized Reasoning. It utilizes Memory-Attention-Composition (MAC) cells—recurrent units that execute a sequence of "read," "control," and "write" operations. In each step, the reasoner focuses on a specific part of the question and queries the visual knowledge base, iteratively refining its understanding to reach a conclusion.

Datasets and Evaluation

The model was rigorously tested on two distinct benchmarks:

  1. SVQA (Synthetic Video Question Answering): Modeled after the CLEVR benchmark, this dataset features complex questions about geometric objects in motion. It is designed to test pure logic and minimize the impact of language shortcuts.
  2. TGIF-QA: A large-scale real-world dataset of 165K QA pairs from animated GIFs, covering tasks like repetition counting, state transitions, and frame-level QA.
Example frames from the SVQA dataset showing moving objects used for logical reasoning tasks.
A real-world video frame from the TGIF-QA dataset, representing the complexity of natural video scenes.

Key Results

The CRN+MAC model achieved a breakthrough on the SVQA dataset with 75.8% accuracy, nearly doubling the previous state-of-the-art of 44.9%. On TGIF-QA, the model set new records for State Transition (78.7%) and Frame QA (59.2%). Ablation studies confirmed that removing the System 2 reasoning module caused SVQA performance to drop significantly (from 75.8% to 49.3%), highlighting the necessity of the "slow" reasoning component for complex tasks.

Insights and Future Directions

The primary takeaway is that disentanglement is key. By separating perception from reasoning, the model handles compositional logic far more effectively. Modeling relations at the clip level rather than the frame level also proves superior for filtering temporal redundancy and identifying long-term relationships.

However, limitations remain. The current implementation lacks explicit motion features (like optical flow), which affected performance on repetition counting tasks. Future work may involve integrating explicit region proposals for objects and actions, as well as exploring bidirectional feedback loops where the "slow" reasoner can guide the "fast" perceiver's focus more dynamically.

Summary updated

Abstract

What does it take to design a machine that learns to answer natural questions about a video? A Video QA system must simultaneously understand language, represent visual content over space-time, and iteratively transform these representations in response to lingual content in the query, and finally arriving at a sensible answer. While recent advances in lingual and visual question answering have enabled sophisticated representations and neural reasoning mechanisms, major challenges in Video QA remain on dynamic grounding of concepts, relations and actions to support the reasoning process. Inspired by the dual-process account of human reasoning, we design a dual process neural architecture, which is composed of a question-guided video processing module (System 1, fast and reactive) followed by a generic reasoning module (System 2, slow and deliberative). System 1 is a hierarchical model that encodes visual patterns about objects, actions and relations in space-time given the textual cues from the question. The encoded representation is a set of high-level visual features, which are then passed to System 2. Here multi-step inference follows to iteratively chain visual elements as instructed by the textual elements. The system is evaluated on the SVQA (synthetic) and TGIF-QA datasets (real), demonstrating competitive results, with a large margin in the case of multi-step reasoning.
BibTeX
@misc{le2019neural,
  author = {Thao Minh Le and Vuong Le and Svetha Venkatesh and Truyen Tran},
  title = {{Neural Reasoning, Fast and Slow, for Video Question Answering}},
  year = {2019},
  eprint = {1907.04553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.04553},
}
Figure