Dense but Efficient VideoQA for Intricate Compositional Reasoning

Jihyeon Lee, Wooyoung Kang, Eun-Sol KimWACV 2023

TL;DR

This paper introduces DSR, a novel VideoQA model designed for complex, multi-step reasoning over long videos. DSR tackles the challenge of efficiently processing dense visual information by employing a deformable attention mechanism for selective feature sampling, guided by question semantics, and a dependency attention module to better understand intricate question structures.

What Problem Does This Paper Address?

Video Question Answering (VideoQA) aims to answer questions about video content. While image-based VQA has seen significant progress, VideoQA, especially for long videos with complex events and their interrelationships, remains a challenge. Existing datasets often feature short clips and simple questions, which can be addressed with less sophisticated methods. However, real-world videos contain intricate compositional structures that require understanding both spatial and temporal dynamics across extended periods. Current VideoQA models struggle with this due to the computational cost of processing dense video data and the difficulty in interpreting complex, multi-step reasoning questions. This paper addresses the need for a VideoQA model that can efficiently handle long videos and complex questions requiring a deep understanding of spatio-temporal relationships.

Methods

DSR is built upon a transformer architecture and introduces two main novel components:

  • Deformable Sampling Module: This module addresses the computational burden of processing dense video features. Instead of processing all frames, it adaptively samples a subset of informative visual features from a dense spatio-temporal feature map. This sampling is conditional on the question's semantics, meaning the model learns to focus on visual elements most relevant to the question. This is achieved through a 3D Conditional Deformable Attention (CDA) mechanism that learns sampling offsets and weights, allowing for a much smaller set of visual tokens to be fed into the cross-modal transformer.
  • Dependency Attention Module: To better understand complex questions that often involve intricate grammatical structures and require multi-step reasoning, DSR incorporates a dependency attention module. This module leverages the dependency parse tree of the question to guide the attention mechanism within the language encoder. By forcing specific attention heads to focus on the grammatical relationships between words, the model can better capture the hierarchical structure of the question, leading to more accurate reasoning.

Additionally, DSR incorporates a global context feature by pooling the sampled local visual features with global visual features, aiming to provide a more comprehensive understanding. Regularization techniques like Soft Orthogonality (SO), Maximal Coding Rate (MCR), and contrastive loss are explored to ensure diversity in the sampled visual tokens and prevent collapsed representations.

Datasets

The primary dataset used for evaluation is Action Genome Question Answering (AGQA). AGQA is highlighted as a particularly challenging benchmark due to its long video clips (average 30 seconds) and questions that inherently require spatio-temporal reasoning. The authors use a balanced, novel composition, and more compositional version of AGQA.

The paper also reports results on other datasets:

  • MSRVTT-QA: A dataset with shorter videos (average 15 seconds) and automatically generated questions, primarily requiring spatial reasoning.
  • TGIF-QA: A dataset of short GIF videos (average 3 seconds) with questions requiring spatial reasoning.
  • TVQA: A dataset of longer video clips (60-90 seconds) with multiple-choice questions.

Key Insights and Ideas

  • Dense is Better (with efficiency): Contrary to some prior work that favored sparse sampling for efficiency, DSR demonstrates that dense visual features, when sampled intelligently, are crucial for complex spatio-temporal reasoning. The key is efficiently sampling from this dense representation.
  • Question-Guided Sampling is Powerful: Conditioning the visual feature sampling on the question's semantics is a critical insight. This allows the model to dynamically focus its attention on the most relevant parts of the video, significantly improving efficiency and accuracy.
  • Linguistic Structure Matters for Reasoning: The success of the dependency attention module highlights that understanding the grammatical structure of a question is not just about word meanings but also about how those meanings are related, which is vital for multi-step reasoning.
  • Global and Local Context Synergy: Combining global visual context with locally sampled, question-relevant features leads to better performance, suggesting that a holistic view alongside focused attention is beneficial.

Key Results

  • On the AGQA dataset, DSR consistently outperforms state-of-the-art baselines, including ClipBERT, achieving significant improvements, especially on the "novel composition" and "more composition" subsets, which are designed to test generalization and reasoning ability. For instance, DSR achieves 65.92% accuracy on the "Binary" type of AGQA's "Full Balanced" subset, outperforming ClipBERT's 63.83%.
  • On MSRVTT-QA and TGIF-QA, DSR achieves comparable scores to ClipBERT, demonstrating its effectiveness even on tasks that don't heavily rely on temporal reasoning.
  • In TVQA, DSR achieves state-of-the-art results in the V+Q setting (using only video and questions, not subtitles).
  • The paper shows that QA accuracy increases with the number of frames considered, but DSR achieves higher accuracy with significantly fewer visual tokens (e.g., 57 vs. 392) compared to sparse sampling methods, demonstrating its efficiency.

Limitations and Future Work

  • Sampling Diversity Regularization: While Soft Orthogonality (SO) regularization performed best, the paper notes that Maximal Coding Rate (MCR) regularization showed high variance in gradients, potentially due to the logdet operator. Contrastive loss also showed lower accuracy, possibly due to small batch sizes. Further exploration into robust regularization techniques could be beneficial.
  • Computational Cost: Although DSR is designed for efficiency, processing long videos and complex transformers still incurs computational costs. The paper mentions that the performance gain is not solely due to increased parameters, but further optimization for real-time applications might be a future direction.
  • Generalization to Other Modalities: The current work focuses on video and text. Future work could explore extending these techniques to other modalities or more complex reasoning tasks.
  • Hyperparameter Tuning: The paper mentions that the number of sampled visual features (object queries) needs careful tuning, as too many can lead to noisy features. Further research into adaptive sampling strategies based on video and question complexity could be explored.

Summary updated

BibTeX
@inproceedings{lee2023dense,
  author = {Jihyeon Lee and Wooyoung Kang and Eun-Sol Kim},
  title = {{Dense but Efficient VideoQA for Intricate Compositional Reasoning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure