Learning to Reason with Relational Video Representation for Question Answering

Thao Minh Le, Vuong Le, Svetha Venkatesh, Truyen Tran

Abstract

Designing machines that can answer natural language questions about video requires a system to simultaneously understand language, represent visual content over space-time, and iteratively transform these representations. Inspired by the dual-process account of human reasoning, we propose a neural architecture composed of a question-guided video processing module (System 1, fast and reactive) followed by a generic reasoning module (System 2, slow and deliberative). System 1 uses a hierarchical Clip-based Relational Network (CRN) to encode visual patterns about objects, actions, and relations. System 2 uses a Memory-Attention-Composition (MAC) network to perform multi-step inference, chaining visual elements as instructed by the query. Our system demonstrates competitive results on the SVQA and TGIF-QA datasets, particularly in complex multi-step reasoning tasks.

Introduction

Answering natural language questions about video is significantly more challenging than static image QA. A successful system must track objects through space-time, recognize actions, and understand temporal relationships (e.g., "What happened after the man sat down?"). Most existing Video QA models "entangle" pattern recognition and reasoning, often relying on black-box recurrent neural networks (RNNs). These systems frequently fall prey to "linguistic bias," guessing answers based on question wording rather than visual evidence.

The Dual-Process Architecture

To address these challenges, we separate visual perception from logical reasoning, mimicking the dual-process model of human cognition.

System 1: Clip-based Relational Network (CRN)

Instead of treating a video as a flat sequence of frames, the CRN builds a hierarchy. It divides the video into clips and uses temporal attention to extract relevant features guided by the question. A relational network then models $k$-order relations between these clips, capturing both short-term actions and long-term temporal dependencies efficiently.

System 2: MAC Network

The output of the CRN serves as a "knowledge base" for the Memory-Attention-Composition (MAC) network. As a generic reasoning engine, the MAC network iteratively performs "Read," "Control," and "Write" operations to chain together visual elements. This allows for complex, compositional reasoning, such as identifying an object based on its relationship to another previously identified object.

Experimental Setup and Datasets

We evaluated the model on two distinct benchmarks:

  1. SVQA (Synthetic Video Question Answering): Designed to test multi-step reasoning with minimal linguistic bias, featuring complex logical structures.
  2. TGIF-QA: A large-scale real-world dataset containing 165K QA pairs from 72K animated GIFs, testing repetition counting, action recognition, and state transitions.
Example frames from the SVQA dataset, requiring multi-step reasoning about object relations over time.

Results and Analysis

The dual-process approach yielded significant performance gains:

  • SVQA: The CRN+MAC model achieved 75.8% overall accuracy, a massive jump from the previous best baseline of 44.9%.
  • TGIF-QA: The model set new state-of-the-art records for State Transition (78.7%) and Frame QA (59.2%).
Real-world video data like TGIF-QA involves complex human actions and state transitions.

Ablation studies confirmed that both systems are essential. Removing the multi-step reasoning (MAC) or the hierarchical video modeling (CRN) significantly degraded performance, proving that disentangling vision from logic is key to handling complex Video QA tasks.

Conclusion and Future Work

By separating the "vision" part (CRN) from the "thinking" part (MAC), our model becomes more modular and resilient to linguistic bias. While the model excels at logical reasoning using appearance features, future work could incorporate explicit motion features (like C3D) to improve action counting. Additionally, exploring "active" perception—where System 2 can direct System 1 to re-examine specific video segments—could further resolve visual ambiguities.

Summary updated

Abstract

What does it take to design a machine that learns to answer natural questions about a video? A Video QA system must simultaneously understand language, represent visual content over space-time, and iteratively transform these representations in response to lingual content in the query, and finally arriving at a sensible answer. While recent advances in lingual and visual question answering have enabled sophisticated representations and neural reasoning mechanisms, major challenges in Video QA remain on dynamic grounding of concepts, relations and actions to support the reasoning process. Inspired by the dual-process account of human reasoning, we design a dual process neural architecture, which is composed of a question-guided video processing module (System 1, fast and reactive) followed by a generic reasoning module (System 2, slow and deliberative). System 1 is a hierarchical model that encodes visual patterns about objects, actions and relations in space-time given the textual cues from the question. The encoded representation is a set of high-level visual features, which are then passed to System 2. Here multi-step inference follows to iteratively chain visual elements as instructed by the textual elements. The system is evaluated on the SVQA (synthetic) and TGIF-QA datasets (real), demonstrating competitive results, with a large margin in the case of multi-step reasoning.
BibTeX
@misc{le2019learning,
  author = {Thao Minh Le and Vuong Le and Svetha Venkatesh and Truyen Tran},
  title = {{Learning to Reason with Relational Video Representation for Question Answering}},
  year = {2019},
  eprint = {1907.04553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.04553},
}
Figure