TL;DR
Current Video Question Answering (VQA) models struggle with complex reasoning because training datasets are overly focused on major events (like a car crash) while ignoring basic scene context (like vehicle colors or directions). To fix this, the authors introduce FIQ, a framework that automatically generates fundamental, descriptive Q&A pairs to enrich training data, alongside a novel VQ-CAlign module that tightly fuses question embeddings with visual features. This combination forces models to understand the broader scene context, achieving state-of-the-art performance on complex traffic reasoning tasks.
What Problem Does This Paper Address?
When humans watch a video, we naturally track objects, their attributes (color, shape), and their trajectories over time. However, existing VQA datasets primarily feature "event-centric" annotations. For example, in a traffic video showing a collision, the dataset might only ask, "Did an accident happen?"
As a result, VQA models become "lazy"—they focus exclusively on the few frames where the event occurs and ignore the preceding or subsequent context. They fail to grasp fundamental details like which vehicles were involved or what happened right before the crash. This fragmented understanding severely limits a model's ability to perform higher-level reasoning, such as causal analysis or future forecasting.
Methods
The proposed FIQ (Fundamental Question Generation with the Integration of Question Embeddings) framework tackles this problem using a two-pronged approach: data enrichment and architectural alignment.
- Fundamental Question Generation:
- The pipeline uses a video-language model (VideoChat2) to extract detailed textual descriptions of low-level features (colors, shapes) and high-level features (motions, temporal order).
- Language models (T5 or GPT-4o-mini) convert these descriptions into diverse Q&A pairs (e.g., "Are there any non-motorized vehicles occupying the driveway?").
- To fit the multi-choice format of standard benchmarks, negative answer options are randomly sampled from other videos in the dataset. These generated pairs are mixed into the training set.
- VQ-CAlign Module:
- Introducing general descriptive data can sometimes cause a model to lose track of task-specific directions. To prevent this, the authors introduce the VQ-CAlign module.
- This module uses self-attention on visual features, followed by cross-attention between the visual features (acting as the query) and the specific question embeddings (acting as the key and value). This ensures that the model focuses on visual regions directly relevant to the question being asked.
- Visual Representation Alignment:
- Finally, an Ans-Decoder fuses the mixed visual-textual features with historical video data to predict the final answer based on cosine similarity.
Datasets
The authors evaluated their approach on SUTD-TrafficQA, a challenging dataset consisting of 10,080 videos and 62,535 human-annotated QA pairs. The dataset is specifically designed to evaluate six distinct reasoning tasks in complex traffic scenarios:
- Basic Understanding (B)
- Event Forecasting (F)
- Reverse Reasoning (R)
- Counterfactual Inference (C)
- Introspection (I)
- Attribution (A)
Key Insights and Ideas
- Basic Questions Unlock Complex Reasoning: The most striking finding is that training a model on simple, fundamental questions (like object counts or colors) significantly boosts its performance on highly complex tasks like Event Forecasting and Reverse Reasoning. By learning to track basic attributes over time, the model develops a more robust causal understanding of the video.
- Context is Temporal: Event-centric annotations cause models to ignore temporal dynamics. Incorporating fundamental scene descriptions forces the model to look at the entire video timeline, establishing better temporal relations between frames.
- LLMs Make Superior Data Augmenters: Utilizing GPT-4o-mini to generate the fundamental Q&A pairs yielded better downstream VQA performance than using smaller language models like T5, highlighting the value of strong pretrained priors for scene description refinement.
Key Results
FIQ achieved a new state-of-the-art average accuracy of 48.4% on the SUTD-TrafficQA dataset, outperforming the previous best baseline, Tem-Adapter (46.0%).
The ablation studies highlighted the individual impact of each component:
- Simply adding the VQ-CAlign module improved accuracy on Event Forecasting from 37.2% to 46.1%.
- Integrating GPT-generated fundamental questions pushed the performance even further, yielding notable gains in Reverse Reasoning (+6.7%) and Introspection (+4.7%) compared to the baseline.
Additionally, the authors noted that models trained with the FIQ framework demonstrated rapid convergence, stabilizing around epoch 20.
Limitations and Future Work
While FIQ shows impressive gains, the authors note that the Counterfactual Inference (C) task saw minimal improvement. This is because counterfactual tasks require reasoning about hypothetical scenarios not depicted in the video, meaning that better grounding in the actual visual scene details doesn't fully solve the problem.
For future work, the authors plan to explore generating entirely new datasets where the question information itself is more deeply integrated inside the dataset as answer candidates to further challenge and improve multimodal reasoning.
Summary updated