ECIS-VQG: Generation of Entity-centric Information-seeking Questions from Videos

Arpan Phukan, Manish Gupta, Asif Ekbal

Abstract

Current video question generation (QG) systems primarily focus on object-centric queries, which often lack the specificity required for real-world information-seeking tasks. This paper introduces ECIS-VQG, a framework designed to generate Entity-centric Information-seeking (ECIS) questions. By leveraging a multimodal Transformer-based architecture and a novel contrastive loss function, ECIS-VQG produces "self-complete" questions that incorporate specific entities like products, places, or people. To facilitate this research, we present VIDEOQUESTIONS, a manually annotated dataset of over 2,200 questions. Our results demonstrate that ECIS-VQG outperforms state-of-the-art large language models (LLMs) such as GPT-4o in generating contextually relevant and fluent information-seeking questions.

Introduction

Most existing video QG systems are limited to identifying common objects or attributes within a frame (e.g., "What is the person holding?"). While useful for basic scene understanding, these systems fail to address the needs of search engine modules or educational tools where users seek specific information about entities.

Example of video content from the VIDEOQUESTIONS dataset, where entities like specific street food locations provide the basis for information-seeking questions.

The challenge in generating ECIS questions lies in three key areas: identifying "question-worthy" segments, accurately linking visual information to entities, and integrating multimodal signals (transcripts, titles, frames) into a "self-complete" question. A self-complete question is one that can be understood without viewing the video, such as "Is the street food at Khaja Ghar affordable?" rather than a vague "Is the food cheap?"

Methods

The ECIS-VQG framework employs a two-stage pipeline to transform video content into high-quality questions.

The ECIS-VQG architecture, featuring a BERT-based chapter classifier, multimodal signal integration (textual and visual), and a contrastive loss training objective.

1. Chapter Title Classification

The system first uses a BERT-based classifier to categorize video chapter titles. Titles are classified as Useless (UL), Self-Complete Questions (SCQ), Not Self-Complete Questions (NSCQ), or Not Self-Complete Phrases (NSCP). Only NSCQ and NSCP types are processed further, as they represent segments requiring context-aware generation.

2. Multimodal Question Generation

The core generation module is a fine-tuned encoder-decoder model (BART or T5). It integrates several input streams:

  • Textual Signals: Video titles, chapter titles, and transcripts.
  • Visual Signals: Frame captions generated via BLIP and visual embeddings from CLIP/ResNeXt.
  • Contextual Refinement: To handle noisy transcripts, GPT-3.5-Turbo is utilized to summarize transcripts and frame captions into a coherent context paragraph before final processing.

3. Contrastive Loss

To prevent the model from defaulting to generic, "safe" questions, the authors implemented a contrastive loss function. This objective encourages the model to maximize the distance between the generated ECIS question and a generic baseline question, forcing a focus on specific entity details.

The VIDEOQUESTIONS Dataset

The researchers introduced VIDEOQUESTIONS, a benchmark consisting of 2,265 manually annotated ECIS questions derived from 411 YouTube videos. The dataset spans diverse domains, including technology, food, and travel, focusing on content where entities are central to the narrative.

Results and Discussion

ECIS-VQG achieved significant performance gains over traditional baselines and modern LLMs. The BART-based model with contrastive loss reached a BLEU-1 of 71.3 and a ROUGE-L of 78.6. In head-to-head evaluations, the fine-tuned model outperformed zero-shot GPT-4o and Llama 3-8B, which frequently struggled with hallucinations or failed to produce truly self-complete questions. Human evaluations further confirmed the high quality of the output, with expert ratings for context relevance (3.91/4) and fluency (3.99/4).

Conclusion

ECIS-VQG represents a significant step toward more useful video-based information retrieval. By focusing on entities and self-completeness, the framework addresses real-world user needs. Future work will focus on expanding the dataset to multiple languages and improving temporal reasoning for complex queries.

Summary updated

Abstract

Previous studies on question generation from videos have mostly focused on generating questions about common objects and attributes and hence are not entity-centric. In this work, we focus on the generation of entity-centric information-seeking questions from videos. Such a system could be useful for video-based learning, recommending ``People Also Ask'' questions, video-based chatbots, and fact-checking. Our work addresses three key challenges: identifying question-worthy information, linking it to entities, and effectively utilizing multimodal signals. Further, to the best of our knowledge, there does not exist a large-scale dataset for this task. Most video question generation datasets are on TV shows, movies, or human activities or lack entity-centric information-seeking questions. Hence, we contribute a diverse dataset of YouTube videos, VideoQuestions, consisting of 411 videos with 2265 manually annotated questions. We further propose a model architecture combining Transformers, rich context signals (titles, transcripts, captions, embeddings), and a combination of cross-entropy and contrastive loss function to encourage entity-centric question generation. Our best method yields BLEU, ROUGE, CIDEr, and METEOR scores of 71.3, 78.6, 7.31, and 81.9, respectively, demonstrating practical usability. We make the code and dataset publicly available. https://github.com/thePhukan/ECIS-VQG
BibTeX
@inproceedings{phukan2024ecis,
  author = {Arpan Phukan and Manish Gupta and Asif Ekbal},
  title = {{ECIS-VQG: Generation of Entity-centric Information-seeking Questions from Videos}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.798},
}
Figure