Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering

Ting Yu, Kunhao Fu, Shuhui Wang, Qingming Huang, Jun Yu

Abstract

Video Question Answering (VideoQA) requires sophisticated reasoning over spatial entities and temporal actions. While Video-Language Foundation Models (VFMs) possess broad cross-modal knowledge, they often struggle with the fine-grained "detective work" needed for complex queries. We introduce HeurVidQA, a framework that treats VFMs as implicit knowledge engines and prompts them with domain-specific heuristics. By generating probabilistic "hints" for actions and entities and using them as supervisory signals, HeurVidQA achieves state-of-the-art results on challenging benchmarks like NExT-QA and SUTD-TrafficQA while maintaining high parameter efficiency.

Introduction

Standard Video-Language Foundation Models (VFMs) like CLIP or ALPRO excel at general video-text matching but often fail at the nuanced demands of VideoQA. Answering a question such as "How did the bride react when the lady in grey hugged her?" requires more than just recognizing a wedding scene; it necessitates pinpointing specific entities and actions across space and time.

The core challenge lies in bridging the gap between general pre-trained knowledge and the specific "detective work" required for VideoQA. Current models often miss these fine-grained cues because their training objectives are too broad. HeurVidQA addresses this by introducing a heuristic-based guidance system that directs the model's attention to critical domain-specific details.

Figure 1: Comparison between standard VFM approaches (1) and the proposed HeurVidQA framework (2), highlighting the use of action-entity heuristics.

Methodology: The HeurVidQA Framework

The HeurVidQA framework operates through a three-stage pipeline designed to distill fine-grained knowledge into the reasoning process.

1. Data Engineering

The process begins by extracting a domain-specific vocabulary—typically the top 1,000 most frequent nouns and verbs—from the target dataset. Videos are then prepared using strategic spatial and temporal cropping to ensure the model can focus on relevant regions and segments.

2. Entity-Action Heuristics Generation (EAPrompter)

The "knowledge engine" of the system is the EAPrompter, which consists of two specialized modules:

  • Action Prompter (AP): Analyzes temporal dynamics to identify verbs and action sequences.
  • Entity Prompter (EP): Focuses on spatial regions to identify key nouns and objects.

These modules generate "heuristics"—probability distributions over the domain vocabulary—that serve as expert hints about the video's content.

3. Heuristic Boosting Answer Inference

Rather than treating these heuristics as simple inputs, HeurVidQA uses them as supervisory targets. This is achieved through two novel loss functions: Temporal Action Modeling (TAM) and Spatial Entity Modeling (SEM). These losses force the model to internalize action sequences and spatial relationships. A Dynamic Gating mechanism further refines the process by adjusting the influence of action versus entity prompts based on the specific question being asked.

Figure 2: The complete HeurVidQA architecture, detailing the flow from data engineering to heuristic generation and final answer inference.

Experimental Results

HeurVidQA was evaluated across several major VideoQA benchmarks, demonstrating superior performance and efficiency.

  • NExT-QA: The model achieved 60.9% accuracy on the validation set, outperforming the ALPRO baseline by 1.0%. It maintained this lead on the test set (59.9%), surpassing much larger models.
  • SUTD-TrafficQA: In complex traffic scenarios, the model showed a 6.7% improvement in "Introspection" questions, highlighting its ability to handle causal reasoning in high-stakes environments.
  • ATP-hard: On this subset designed to test true video-level understanding, HeurVidQA scored 47.3%, significantly higher than the 45.3% baseline.

Notably, HeurVidQA achieves these results using only 16 frames, compared to the 100+ frames required by some competitors. It also remains parameter-efficient, utilizing 237M parameters—significantly fewer than models like JustAsk (600M).

Conclusion

HeurVidQA demonstrates that the power of foundation models can be unlocked for specialized tasks like VideoQA through targeted, heuristic-based prompting. By using domain-specific actions and entities as supervisory signals, the framework bridges the gap between general knowledge and fine-grained reasoning. Future work will explore the integration of knowledge graphs and adaptive prompt learning to further enhance the model's abstract reasoning capabilities.

Summary updated

Abstract

Video Question Answering (VideoQA) represents a crucial intersection between video understanding and language processing, requiring both discriminative unimodal comprehension and sophisticated cross-modal interaction for accurate inference. Despite advancements in multi-modal pre-trained models and video-language foundation models, these systems often struggle with domain-specific VideoQA due to their generalized pre-training objectives. Addressing this gap necessitates bridging the divide between broad cross-modal knowledge and the specific inference demands of VideoQA tasks. To this end, we introduce HeurVidQA, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning. By delivering fine-grained heuristics, we improve the model's ability to identify and interpret key entities and actions, thereby enhancing its reasoning capabilities. Extensive evaluations across multiple VideoQA datasets demonstrate that our method significantly outperforms existing models, underscoring the importance of integrating domain-specific knowledge into video-language models for more accurate and context-aware VideoQA.
BibTeX
@article{yu2024prompting,
  author = {Ting Yu and Kunhao Fu and Shuhui Wang and Qingming Huang and Jun Yu},
  title = {{Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  doi = {10.1109/tcsvt.2024.3475510},
}
Figure