Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering

Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, Yue Gao

Introduction

Visual Question Answering (VQA) has seen significant success in the domain of static images, where models primarily focus on spatial features to identify objects and their attributes. However, Video Question Answering (Video QA) presents a more complex challenge. Unlike static images, video requires a model to navigate two dimensions simultaneously: Space (e.g., "Where is the person?") and Time (e.g., "What did they do after opening the door?").

Existing research often treats the input question as a monolithic entity, failing to leverage the distinct linguistic cues that point to spatial or temporal information. The QueST (Question-Guided Spatio-Temporal Contextual Attention) network addresses this gap by adopting a "Divide and Conquer" strategy, explicitly separating a question's intent into its spatial and temporal components to guide more precise video understanding.

Conceptual overview of the Divide and Conquer approach: splitting a question into spatial and temporal components to guide attention.

The QueST Framework

The QueST architecture is built around two primary modules designed to handle the linguistic and visual complexities of video data.

1. Video-Guided Question Attention (VGQA): The "Divide" Stage

Instead of generating a single question embedding, the VGQA module produces two specialized representations: a spatial question embedding and a temporal question embedding.

To ensure these embeddings are truly distinct, the researchers implemented a Diversity Loss function. This mechanism forces the model to attend to different words for each embedding—for instance, focusing on "man" and "door" for spatial context, while highlighting "after" and "opening" for temporal context. This linguistic specialization allows the model to create more effective "guides" for the subsequent visual attention stages.

2. Contextual Attention Block (CAB): The "Conquer" Stage

The visual features of the video are processed through two specialized sub-blocks, each guided by the corresponding question embedding:

  • Spatial CAB (SCAB): This block uses the spatial question embedding to highlight relevant regions within individual video frames, effectively locating the "where."
  • Multi-Scale Temporal CAB (MS-TCAB): This novel component uses 1D convolutional layers with varying dilation rates. This multi-scale approach allows the model to capture actions occurring at different temporal resolutions—from rapid gestures to long-duration state changes—guided by the temporal question embedding.
The overall architecture of the QueST network, showing the flow from question/video features to the specialized attention blocks and final fusion.

Experimental Results

The QueST model was evaluated on three major Video QA benchmarks: TGIF-QA, MSRVTT-QA, and MSVD-QA. The results demonstrated state-of-the-art (SOTA) performance across multiple task types:

  • TGIF-QA: QueST outperformed previous leaders by 2.0% to 4.0% in accuracy for action, transition, and frame-based QA tasks.
  • MSRVTT-QA & MSVD-QA: The model achieved overall accuracy boosts of 1.6% and 2.4%, respectively, over previous best-performing models.
  • Ablation Studies: The authors confirmed that the "Divide" strategy was critical; removing the Diversity Loss or the specialized Word Attention module resulted in a measurable drop in performance, proving that specialized embeddings are key to superior video understanding.

Conclusion and Future Work

By explicitly separating the spatial and temporal cues within a question, the QueST network provides a more nuanced and effective way for AI to "watch" and interpret video content. While the model excels at identifying actions and transitions, it still faces challenges in precise Counting tasks (e.g., "How many times does the girl clap?"), where it was slightly edged out by iterative logic models like HME.

Future work aims to integrate more complex reasoning modules to bridge the gap between spatio-temporal attention and high-level cognitive deduction, potentially solving the remaining hurdles in complex video reasoning.

Summary updated

Abstract

Understanding questions and finding clues for answers are the key for video question answering. Compared with image question answering, video question answering (Video QA) requires to find the clues accurately on both spatial and temporal dimension simultaneously, and thus is more challenging. However, the relationship between spatio-temporal information and question still has not been well utilized in most existing methods for Video QA. To tackle this problem, we propose a Question-Guided Spatio-Temporal Contextual Attention Network (QueST) method. In QueST, we divide the semantic features generated from question into two separate parts: the spatial part and the temporal part, respectively guiding the process of constructing the contextual attention on spatial and temporal dimension. Under the guidance of the corresponding contextual attention, visual features can be better exploited on both spatial and temporal dimensions. To evaluate the effectiveness of the proposed method, experiments are conducted on TGIF-QA dataset, MSRVTT-QA dataset and MSVD-QA dataset. Experimental results and comparisons with the state-of-the-art methods have shown that our method can achieve superior performance.
BibTeX
@inproceedings{jiang2020divide,
  author = {Jianwen Jiang and Ziqiang Chen and Haojie Lin and Xibin Zhao and Yue Gao},
  title = {{Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6766},
}
Figure