FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering

Ju-Young Oh, Ho-Joong Kim, Seong-Whan LeearXiv 2025 · arXiv

TL;DR

Current Video Question Answering (VQA) models struggle with complex reasoning because training datasets are overly focused on major events (like a car crash) while ignoring basic scene context (like vehicle colors or directions). To fix this, the authors introduce FIQ, a framework that automatically generates fundamental, descriptive Q&A pairs to enrich training data, alongside a novel VQ-CAlign module that tightly fuses question embeddings with visual features. This combination forces models to understand the broader scene context, achieving state-of-the-art performance on complex traffic reasoning tasks.

What Problem Does This Paper Address?

When humans watch a video, we naturally track objects, their attributes (color, shape), and their trajectories over time. However, existing VQA datasets primarily feature "event-centric" annotations. For example, in a traffic video showing a collision, the dataset might only ask, "Did an accident happen?"

As a result, VQA models become "lazy"—they focus exclusively on the few frames where the event occurs and ignore the preceding or subsequent context. They fail to grasp fundamental details like which vehicles were involved or what happened right before the crash. This fragmented understanding severely limits a model's ability to perform higher-level reasoning, such as causal analysis or future forecasting.

The limitations of event-centric datasets vs. fundamental scene information.

Methods

The proposed FIQ (Fundamental Question Generation with the Integration of Question Embeddings) framework tackles this problem using a two-pronged approach: data enrichment and architectural alignment.

Overall architecture of the FIQ framework.
  1. Fundamental Question Generation:
  2. The pipeline uses a video-language model (VideoChat2) to extract detailed textual descriptions of low-level features (colors, shapes) and high-level features (motions, temporal order).
  3. Language models (T5 or GPT-4o-mini) convert these descriptions into diverse Q&A pairs (e.g., "Are there any non-motorized vehicles occupying the driveway?").
  4. To fit the multi-choice format of standard benchmarks, negative answer options are randomly sampled from other videos in the dataset. These generated pairs are mixed into the training set.
  5. VQ-CAlign Module:
  6. Introducing general descriptive data can sometimes cause a model to lose track of task-specific directions. To prevent this, the authors introduce the VQ-CAlign module.
  7. This module uses self-attention on visual features, followed by cross-attention between the visual features (acting as the query) and the specific question embeddings (acting as the key and value). This ensures that the model focuses on visual regions directly relevant to the question being asked.
  8. Visual Representation Alignment:
  9. Finally, an Ans-Decoder fuses the mixed visual-textual features with historical video data to predict the final answer based on cosine similarity.

Datasets

The authors evaluated their approach on SUTD-TrafficQA, a challenging dataset consisting of 10,080 videos and 62,535 human-annotated QA pairs. The dataset is specifically designed to evaluate six distinct reasoning tasks in complex traffic scenarios:

  • Basic Understanding (B)
  • Event Forecasting (F)
  • Reverse Reasoning (R)
  • Counterfactual Inference (C)
  • Introspection (I)
  • Attribution (A)

Key Insights and Ideas

  • Basic Questions Unlock Complex Reasoning: The most striking finding is that training a model on simple, fundamental questions (like object counts or colors) significantly boosts its performance on highly complex tasks like Event Forecasting and Reverse Reasoning. By learning to track basic attributes over time, the model develops a more robust causal understanding of the video.
  • Context is Temporal: Event-centric annotations cause models to ignore temporal dynamics. Incorporating fundamental scene descriptions forces the model to look at the entire video timeline, establishing better temporal relations between frames.
  • LLMs Make Superior Data Augmenters: Utilizing GPT-4o-mini to generate the fundamental Q&A pairs yielded better downstream VQA performance than using smaller language models like T5, highlighting the value of strong pretrained priors for scene description refinement.

Key Results

FIQ achieved a new state-of-the-art average accuracy of 48.4% on the SUTD-TrafficQA dataset, outperforming the previous best baseline, Tem-Adapter (46.0%).

Performance comparison of FIQ against state-of-the-art methods on SUTD-TrafficQA.

The ablation studies highlighted the individual impact of each component:

  • Simply adding the VQ-CAlign module improved accuracy on Event Forecasting from 37.2% to 46.1%.
  • Integrating GPT-generated fundamental questions pushed the performance even further, yielding notable gains in Reverse Reasoning (+6.7%) and Introspection (+4.7%) compared to the baseline.
Ablation study showing the impact of VQ-CAlign and LM/LLM data integration.

Additionally, the authors noted that models trained with the FIQ framework demonstrated rapid convergence, stabilizing around epoch 20.

Accuracy curves over training epochs for different FIQ configurations.

Limitations and Future Work

While FIQ shows impressive gains, the authors note that the Counterfactual Inference (C) task saw minimal improvement. This is because counterfactual tasks require reasoning about hypothetical scenarios not depicted in the video, meaning that better grounding in the actual visual scene details doesn't fully solve the problem.

For future work, the authors plan to explore generating entirely new datasets where the question information itself is more deeply integrated inside the dataset as answer candidates to further challenge and improve multimodal reasoning.

Summary updated

Abstract

Video question answering (VQA) is a multimodal task that requires the interpretation of a video to answer a given question. Existing VQA methods primarily utilize question and answer (Q&A) pairs to learn the spatio-temporal characteristics of video content. However, these annotations are typically event-centric, which is not enough to capture the broader context of each video. The absence of essential details such as object types, spatial layouts, and descriptive attributes restricts the model to learning only a fragmented scene representation. This issue limits the model's capacity for generalization and higher-level reasoning. In this paper, we propose a fundamental question generation with the integration of question embeddings for video question answering (FIQ), a novel approach designed to strengthen the reasoning ability of the model by enhancing the fundamental understanding of videos. FIQ generates Q&A pairs based on descriptions extracted from videos, enriching the training data with fundamental scene information. Generated Q&A pairs enable the model to understand the primary context, leading to enhanced generalizability and reasoning ability. Furthermore, we incorporate a VQ-CAlign module that assists task-specific question embeddings with visual features, ensuring that essential domain-specific details are preserved to increase the adaptability of downstream tasks. Experiments on SUTD-TrafficQA demonstrate that our FIQ achieves state-of-the-art performance compared to existing baseline methods.
BibTeX
@misc{oh2025fiq,
  author = {Ju-Young Oh and Ho-Joong Kim and Seong-Whan Lee},
  title = {{FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering}},
  year = {2025},
  eprint = {2507.12816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.12816},
}

Loading related content...

Figure