TL;DR
This paper introduces GP-VQA, a novel approach to Video Question Answering (VideoQA) that enhances temporal reasoning by leveraging video-based scene graphs. By pre-training a model to complete masked video graphs and then using these learned reasoning abilities to interpret question graphs, GP-VQA can better understand video content not explicitly mentioned in the question, leading to improved accuracy.
What Problem Does This Paper Address?
Video Question Answering (VideoQA) aims to answer questions about video content. A key challenge is understanding the temporal dynamics and relationships between objects within a video, especially when the question only hints at certain elements. Existing methods often rely heavily on the question text to guide their understanding of the video, which can lead to them overlooking crucial information present in the video but not mentioned in the question. For instance, if asked "Why is the toddler crying?" and a dog is interacting with the toddler, a question-guided model might miss the dog's role because it's not in the question. This paper addresses this limitation by proposing a video-guided approach that allows the model to reason about the entire video context, not just the question-specified elements.
Methods
The proposed GP-VQA model employs a two-stage approach:
- Pre-training with Masked Graph Completion:
- Video Scene Graph Generation (SGG): The process starts by generating a video scene graph, which represents objects and their relationships within each frame of the video. This is done using existing SGG models.
- Masked Graph Completion: To encourage robust temporal reasoning, random nodes and edges in these video scene graphs are masked. The model is then trained to predict and complete these masked parts. This pretext task forces the model to learn general reasoning abilities about video content, independent of specific questions.
- Prompt-tuning with VideoQA:
- Question Graph Construction: The textual question is converted into a "question graph" by analyzing its semantic dependencies.
- Message Passing: A crucial step involves message passing between the pre-trained video graph and the question graph. This allows the question graph to "inherit" the temporal reasoning capabilities learned during pre-training.
- Graph Prompts: The integrated video-question graph is then used as a "prompt" for a language model, which, with a new QA head, predicts the final answer. This approach ensures that the model's reasoning is grounded in the video's visual and temporal context.
Datasets
The paper evaluates GP-VQA on three challenging VideoQA datasets:
- AGQA v2: An open-ended benchmark for compositional spatial-temporal reasoning.
- NExT-QA: A benchmark for causal and temporal reasoning.
- STAR: A benchmark for situated reasoning.
The authors also mention using the Action Genome dataset for pre-training video scene graphs, which is a large-scale dataset for video understanding.
Key Insights and Ideas
- Video-Guided Reasoning is Crucial: The core insight is that relying solely on question text for video understanding is insufficient. By using video scene graphs as a primary source of information and learning to reason about them, models can achieve a more comprehensive understanding of video content.
- Masked Graph Completion as a Powerful Pretext Task: The masked video graph completion task effectively trains the model to learn general temporal reasoning skills that are transferable to the downstream VideoQA task. It acts as a form of "self-supervision" for temporal understanding.
- Graph Prompts Bridge Pre-training and Fine-tuning: The concept of using the integrated video-question graph as a "prompt" is a clever way to transfer the learned reasoning abilities from the pre-training stage to the VideoQA task, enabling a more seamless integration.
- Flexibility of the Framework: The GP-VQA framework is shown to be flexible, capable of incorporating different video scene graph generation models, demonstrating its adaptability.
Key Results
GP-VQA achieves state-of-the-art (SOTA) performance on all three evaluated datasets (AGQA v2, NExT-QA, and STAR).
- On AGQA v2, GP-VQA shows significant improvements, particularly in relation-action and action recognition types.
- On NExT-QA, despite not being able to pre-train on the dataset itself, GP-VQA still demonstrates superior performance, especially in causal-type questions, outperforming strong baselines like SeVila.
- On STAR, GP-VQA also shows substantial improvements across various metrics, highlighting the effectiveness of the masked video graph completion pre-training. The paper also includes ablation studies that confirm the importance of each component, such as the pre-training strategy and the random masking technique.
Limitations and Future Work
- Limitations:
- The paper notes that for NExT-QA, they couldn't perform masked video graph completion pre-training directly on the dataset, which might have limited the performance gain compared to AGQA v2.
- The performance improvement on NExT-QA is described as "lower" compared to AGQA v2, though still superior to SOTA.
- Future Work:
- The authors suggest exploring semi-supervised or unsupervised graph-based prompt-tuning.
- They aim for practical real-world adaptation, implying a focus on making the model more robust and applicable in diverse scenarios.
Summary updated