Abstract
Current video question generation (QG) systems primarily focus on object-centric queries, which often lack the specificity required for real-world information-seeking tasks. This paper introduces ECIS-VQG, a framework designed to generate Entity-centric Information-seeking (ECIS) questions. By leveraging a multimodal Transformer-based architecture and a novel contrastive loss function, ECIS-VQG produces "self-complete" questions that incorporate specific entities like products, places, or people. To facilitate this research, we present VIDEOQUESTIONS, a manually annotated dataset of over 2,200 questions. Our results demonstrate that ECIS-VQG outperforms state-of-the-art large language models (LLMs) such as GPT-4o in generating contextually relevant and fluent information-seeking questions.
Introduction
Most existing video QG systems are limited to identifying common objects or attributes within a frame (e.g., "What is the person holding?"). While useful for basic scene understanding, these systems fail to address the needs of search engine modules or educational tools where users seek specific information about entities.
The challenge in generating ECIS questions lies in three key areas: identifying "question-worthy" segments, accurately linking visual information to entities, and integrating multimodal signals (transcripts, titles, frames) into a "self-complete" question. A self-complete question is one that can be understood without viewing the video, such as "Is the street food at Khaja Ghar affordable?" rather than a vague "Is the food cheap?"
Methods
The ECIS-VQG framework employs a two-stage pipeline to transform video content into high-quality questions.
1. Chapter Title Classification
The system first uses a BERT-based classifier to categorize video chapter titles. Titles are classified as Useless (UL), Self-Complete Questions (SCQ), Not Self-Complete Questions (NSCQ), or Not Self-Complete Phrases (NSCP). Only NSCQ and NSCP types are processed further, as they represent segments requiring context-aware generation.
2. Multimodal Question Generation
The core generation module is a fine-tuned encoder-decoder model (BART or T5). It integrates several input streams:
- Textual Signals: Video titles, chapter titles, and transcripts.
- Visual Signals: Frame captions generated via BLIP and visual embeddings from CLIP/ResNeXt.
- Contextual Refinement: To handle noisy transcripts, GPT-3.5-Turbo is utilized to summarize transcripts and frame captions into a coherent context paragraph before final processing.
3. Contrastive Loss
To prevent the model from defaulting to generic, "safe" questions, the authors implemented a contrastive loss function. This objective encourages the model to maximize the distance between the generated ECIS question and a generic baseline question, forcing a focus on specific entity details.
The VIDEOQUESTIONS Dataset
The researchers introduced VIDEOQUESTIONS, a benchmark consisting of 2,265 manually annotated ECIS questions derived from 411 YouTube videos. The dataset spans diverse domains, including technology, food, and travel, focusing on content where entities are central to the narrative.
Results and Discussion
ECIS-VQG achieved significant performance gains over traditional baselines and modern LLMs. The BART-based model with contrastive loss reached a BLEU-1 of 71.3 and a ROUGE-L of 78.6. In head-to-head evaluations, the fine-tuned model outperformed zero-shot GPT-4o and Llama 3-8B, which frequently struggled with hallucinations or failed to produce truly self-complete questions. Human evaluations further confirmed the high quality of the output, with expert ratings for context relevance (3.91/4) and fluency (3.99/4).
Conclusion
ECIS-VQG represents a significant step toward more useful video-based information retrieval. By focusing on entities and self-completeness, the framework addresses real-world user needs. Future work will focus on expanding the dataset to multiple languages and improving temporal reasoning for complex queries.
Summary updated