Graph Prompts: Adapting Video Graph for Video Question Answering

Yiming Li, Xiaoshan Yang, Bing-Kun Bao, Changsheng XuIJCAI 2025 · DOI · OpenReview

TL;DR

This paper introduces GP-VQA, a novel approach to Video Question Answering (VideoQA) that enhances temporal reasoning by leveraging video-based scene graphs. By pre-training a model to complete masked video graphs and then using these learned reasoning abilities to interpret question graphs, GP-VQA can better understand video content not explicitly mentioned in the question, leading to improved accuracy.

What Problem Does This Paper Address?

Video Question Answering (VideoQA) aims to answer questions about video content. A key challenge is understanding the temporal dynamics and relationships between objects within a video, especially when the question only hints at certain elements. Existing methods often rely heavily on the question text to guide their understanding of the video, which can lead to them overlooking crucial information present in the video but not mentioned in the question. For instance, if asked "Why is the toddler crying?" and a dog is interacting with the toddler, a question-guided model might miss the dog's role because it's not in the question. This paper addresses this limitation by proposing a video-guided approach that allows the model to reason about the entire video context, not just the question-specified elements.

Methods

The proposed GP-VQA model employs a two-stage approach:

  1. Pre-training with Masked Graph Completion:
    • Video Scene Graph Generation (SGG): The process starts by generating a video scene graph, which represents objects and their relationships within each frame of the video. This is done using existing SGG models.
  2. Masked Graph Completion: To encourage robust temporal reasoning, random nodes and edges in these video scene graphs are masked. The model is then trained to predict and complete these masked parts. This pretext task forces the model to learn general reasoning abilities about video content, independent of specific questions.
  3. Prompt-tuning with VideoQA:
    • Question Graph Construction: The textual question is converted into a "question graph" by analyzing its semantic dependencies.
  4. Message Passing: A crucial step involves message passing between the pre-trained video graph and the question graph. This allows the question graph to "inherit" the temporal reasoning capabilities learned during pre-training.
  5. Graph Prompts: The integrated video-question graph is then used as a "prompt" for a language model, which, with a new QA head, predicts the final answer. This approach ensures that the model's reasoning is grounded in the video's visual and temporal context.

Datasets

The paper evaluates GP-VQA on three challenging VideoQA datasets:

  • AGQA v2: An open-ended benchmark for compositional spatial-temporal reasoning.
  • NExT-QA: A benchmark for causal and temporal reasoning.
  • STAR: A benchmark for situated reasoning.

The authors also mention using the Action Genome dataset for pre-training video scene graphs, which is a large-scale dataset for video understanding.

Key Insights and Ideas

  • Video-Guided Reasoning is Crucial: The core insight is that relying solely on question text for video understanding is insufficient. By using video scene graphs as a primary source of information and learning to reason about them, models can achieve a more comprehensive understanding of video content.
  • Masked Graph Completion as a Powerful Pretext Task: The masked video graph completion task effectively trains the model to learn general temporal reasoning skills that are transferable to the downstream VideoQA task. It acts as a form of "self-supervision" for temporal understanding.
  • Graph Prompts Bridge Pre-training and Fine-tuning: The concept of using the integrated video-question graph as a "prompt" is a clever way to transfer the learned reasoning abilities from the pre-training stage to the VideoQA task, enabling a more seamless integration.
  • Flexibility of the Framework: The GP-VQA framework is shown to be flexible, capable of incorporating different video scene graph generation models, demonstrating its adaptability.

Key Results

GP-VQA achieves state-of-the-art (SOTA) performance on all three evaluated datasets (AGQA v2, NExT-QA, and STAR).

  • On AGQA v2, GP-VQA shows significant improvements, particularly in relation-action and action recognition types.
  • On NExT-QA, despite not being able to pre-train on the dataset itself, GP-VQA still demonstrates superior performance, especially in causal-type questions, outperforming strong baselines like SeVila.
  • On STAR, GP-VQA also shows substantial improvements across various metrics, highlighting the effectiveness of the masked video graph completion pre-training. The paper also includes ablation studies that confirm the importance of each component, such as the pre-training strategy and the random masking technique.

Limitations and Future Work

  • Limitations:
    • The paper notes that for NExT-QA, they couldn't perform masked video graph completion pre-training directly on the dataset, which might have limited the performance gain compared to AGQA v2.
  • The performance improvement on NExT-QA is described as "lower" compared to AGQA v2, though still superior to SOTA.
  • Future Work:
    • The authors suggest exploring semi-supervised or unsupervised graph-based prompt-tuning.
  • They aim for practical real-world adaptation, implying a focus on making the model more robust and applicable in diverse scenarios.

Summary updated

Abstract

Due to the dynamic nature in videos, it is evident that perceiving and reasoning about temporal information are the key focus of Video Question Answering (VideoQA). In recent years, several methods have explored relationship-level temporal modeling with graph-structured video representation. Unfortunately, these methods heavily rely on the question text, thus making it challenging to perceive and reason about video content that is not explicitly mentioned in the question. To address the above challenge, we propose Graph Prompts-based VideoQA (GP-VQA), which adopts a video-based graph structure for enhanced video understanding. The proposed GP-VQA contains two stages, i.e., pre-training and prompt tuning. In pre-training, we define the pretext task that requires GP-VQA to reason about the randomly masked nodes or edges in the video graph, thus prompting GP-VQA to learn the reasoning ability with video-guided information. In prompt-tuning, we organize the textual question into question graph and implement message passing from video graph to question graph, therefore inheriting the video-based reasoning ability from video graph completion to VideoQA. Extensive experiments on various datasets have demonstrated the promising performance of GP-VQA.
BibTeX
@inproceedings{li2025graph,
  author = {Yiming Li and Xiaoshan Yang and Bing-Kun Bao and Changsheng Xu},
  title = {{Graph Prompts: Adapting Video Graph for Video Question Answering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/166},
}
Figure