Abstract
Video Question Answering (VideoQA) requires sophisticated reasoning over spatial entities and temporal actions. While Video-Language Foundation Models (VFMs) possess broad cross-modal knowledge, they often struggle with the fine-grained "detective work" needed for complex queries. We introduce HeurVidQA, a framework that treats VFMs as implicit knowledge engines and prompts them with domain-specific heuristics. By generating probabilistic "hints" for actions and entities and using them as supervisory signals, HeurVidQA achieves state-of-the-art results on challenging benchmarks like NExT-QA and SUTD-TrafficQA while maintaining high parameter efficiency.
Introduction
Standard Video-Language Foundation Models (VFMs) like CLIP or ALPRO excel at general video-text matching but often fail at the nuanced demands of VideoQA. Answering a question such as "How did the bride react when the lady in grey hugged her?" requires more than just recognizing a wedding scene; it necessitates pinpointing specific entities and actions across space and time.
The core challenge lies in bridging the gap between general pre-trained knowledge and the specific "detective work" required for VideoQA. Current models often miss these fine-grained cues because their training objectives are too broad. HeurVidQA addresses this by introducing a heuristic-based guidance system that directs the model's attention to critical domain-specific details.
Methodology: The HeurVidQA Framework
The HeurVidQA framework operates through a three-stage pipeline designed to distill fine-grained knowledge into the reasoning process.
1. Data Engineering
The process begins by extracting a domain-specific vocabulary—typically the top 1,000 most frequent nouns and verbs—from the target dataset. Videos are then prepared using strategic spatial and temporal cropping to ensure the model can focus on relevant regions and segments.
2. Entity-Action Heuristics Generation (EAPrompter)
The "knowledge engine" of the system is the EAPrompter, which consists of two specialized modules:
- Action Prompter (AP): Analyzes temporal dynamics to identify verbs and action sequences.
- Entity Prompter (EP): Focuses on spatial regions to identify key nouns and objects.
These modules generate "heuristics"—probability distributions over the domain vocabulary—that serve as expert hints about the video's content.
3. Heuristic Boosting Answer Inference
Rather than treating these heuristics as simple inputs, HeurVidQA uses them as supervisory targets. This is achieved through two novel loss functions: Temporal Action Modeling (TAM) and Spatial Entity Modeling (SEM). These losses force the model to internalize action sequences and spatial relationships. A Dynamic Gating mechanism further refines the process by adjusting the influence of action versus entity prompts based on the specific question being asked.
Experimental Results
HeurVidQA was evaluated across several major VideoQA benchmarks, demonstrating superior performance and efficiency.
- NExT-QA: The model achieved 60.9% accuracy on the validation set, outperforming the ALPRO baseline by 1.0%. It maintained this lead on the test set (59.9%), surpassing much larger models.
- SUTD-TrafficQA: In complex traffic scenarios, the model showed a 6.7% improvement in "Introspection" questions, highlighting its ability to handle causal reasoning in high-stakes environments.
- ATP-hard: On this subset designed to test true video-level understanding, HeurVidQA scored 47.3%, significantly higher than the 45.3% baseline.
Notably, HeurVidQA achieves these results using only 16 frames, compared to the 100+ frames required by some competitors. It also remains parameter-efficient, utilizing 237M parameters—significantly fewer than models like JustAsk (600M).
Conclusion
HeurVidQA demonstrates that the power of foundation models can be unlocked for specialized tasks like VideoQA through targeted, heuristic-based prompting. By using domain-specific actions and entities as supervisory signals, the framework bridges the gap between general knowledge and fine-grained reasoning. Future work will explore the integration of knowledge graphs and adaptive prompt learning to further enhance the model's abstract reasoning capabilities.
Summary updated