Causal and Temporal Inference in Visual Question Generation by Utilizing Pre-trained Models

Zhanghao Hu, Frank KellerACL 2024

TL;DR

This paper introduces a novel framework for Visual Question Generation (VQG) that leverages pre-trained vision-text models to generate inferential questions about videos, focusing on causal and temporal relationships. By adapting existing models rather than training from scratch, they achieve improved question quality and efficiency, addressing a key gap in VQG research.

What Problem Does This Paper Address?

Visual Question Generation (VQG) is a task that bridges vision and language, aiming to generate questions about visual content. While current VQG models excel at generating fact-based questions from static images, they struggle with inferential questions that require understanding deeper relationships within videos. Specifically, existing methods often fall short in capturing the causal (why things happen) and temporal (what happens next) relationships between events and entities in a video. Furthermore, training large VQG models from scratch is computationally expensive. This paper aims to address these limitations by developing a framework that can generate inferential questions from videos using pre-trained models, making the task more accessible and improving the quality of generated questions.

Methods

The core of the proposed framework involves a multi-modal fusion approach. They utilize pre-trained vision-text matching models (like CLIP, BLIP, BLIP2) as visual encoders and a pre-trained language model (T5) as the question decoder.

Key methodological contributions include:

  • Vision-Text Matching for Inference: Instead of training a visual encoder from scratch, they leverage the powerful representations learned by pre-trained vision-text models. These models are adept at aligning visual and textual information, which is crucial for understanding the nuances of video content.
  • Multi-modal Fusion: Visual features from the video encoder and text embeddings from auxiliary text are concatenated and fed into a language model. This allows the language model to condition its question generation on both visual and textual context.
  • Two-Stage Fine-Tuning: To optimize the alignment between visual and language models, they employ a two-stage fine-tuning strategy. The first stage focuses on feature alignment, while the second stage fine-tunes the entire model end-to-end.
  • Causal and Temporal Inference Abstraction: They explore methods to abstract causal and temporal relationships. This includes:
  • Vision Projection Matrix: Using MLPs to map visual embeddings to a linguistic embedding space.
  • Contradictory Frame Comparison: Comparing consecutive frames to identify key differences that might indicate causal or temporal shifts. They propose both global and local frame comparison strategies.

Datasets

The primary dataset used for training and evaluation is NExT-QA. This dataset is specifically designed for inferential visual question answering, containing approximately 52K diverse questions, with a significant portion being causal (48%), temporal (29%), and descriptive (23%).

Key Insights and Ideas

  • Pre-trained Models are Powerful for VQG: Leveraging existing vision-text matching models significantly boosts performance and reduces computational cost compared to training from scratch.
  • Consistency is Key for Inference: Maintaining consistent relationships between input frames during both training and inference is crucial for the language model to effectively infer causal and temporal relationships. Random frame selection or inconsistent frame processing can degrade performance.
  • Global Frame Comparison Outperforms Local: The global frame comparison method, which considers broader frame relationships, yields better results than local frame comparison, suggesting that a more holistic view of frame interactions is beneficial for inference.
  • Two-Stage Tuning Enhances Performance: A two-stage fine-tuning approach, focusing first on feature alignment and then on end-to-end optimization, leads to improved question generation quality.

Key Results

The proposed framework, utilizing pre-trained models like CLIP and T5, demonstrates strong performance on the NExT-QA dataset.

  • Improved Question Quality: The method successfully guides pre-trained language models to generate more insightful, inferential questions compared to baseline models.
  • CLIP as a Strong Visual Encoder: CLIP emerged as a top-performing visual encoder, showing excellent results in ROUGEL and grounding metrics.
  • T5 Large Outperforms T5 Small: Larger language models (T5 Large) generally perform better, especially with the two-stage tuning strategy.
  • Grounding Metrics are Important: The authors introduce new grounding metrics (Precision, Recall, F1-score) that focus on content-bearing words, proving effective in evaluating the relevance of generated questions to the video content.
  • Global Frame Comparison shows promise: While slightly behind direct vision embedding concatenation, the global frame comparison method showed a substantial 20% boost in causal and temporal questions compared to random selection.

Limitations and Future Work

  • Model Size: The study primarily uses T5 encoder-decoder models due to computational constraints and limited GPUs. Future work could explore larger parameter models and decoder-only structures for potentially better performance.
  • Separate Causal/Temporal Research: The authors suggest that future research could separately investigate causal and temporal relationships between entities within videos for deeper understanding.
  • Negative Methods: The paper explored contrastive learning and visual-semantic arithmetic methods as negative approaches. While showing some promise, their effectiveness was limited, suggesting avenues for further refinement, particularly in expanding negative sample pools and exploring more sophisticated frame relationship techniques.
  • Consistent Frame Relationships: The importance of consistent frame relationships is highlighted, suggesting future work could focus on developing methods to improve this consistency.

Summary updated

BibTeX
@inproceedings{hu2024causal,
  author = {Zhanghao Hu and Frank Keller},
  title = {{Causal and Temporal Inference in Visual Question Generation by Utilizing Pre-trained Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure