Category-Guided Visual Question Generation (Student Abstract)

Hongfei Liu, Jiali Chen, Wenhao Fang, Jiayuan Xie, Yi CaiAAAI 2023 · DOI

Abstract

Traditional Visual Question Generation (VQG) models often produce generic, repetitive questions. This paper introduces Category-Guided Visual Question Generation (CGVQG), a framework that utilizes specific scene categories to drive more diverse and contextually relevant questions. By employing a TF-IDF-based weighting system to link visual objects to question types, the model achieves superior diversity compared to unconstrained baselines.

Introduction

Visual Question Generation (VQG) aims to enable AI systems to ask meaningful questions about visual content. While useful for interactive AI and education, current models suffer from a "boringness" problem, frequently defaulting to safe, generic questions like "What is this?" or "Where is the person?" This occurs because models lack a specific target or category to focus their inquiry, failing to capture the nuanced content of a scene.

Methodology: The CGVQG Framework

The proposed CGVQG model addresses the diversity challenge by breaking the generation process into three distinct stages:

  1. Visual Encoder: Utilizing Faster R-CNN for object detection and VisualBert for relational understanding, the model applies a TF-IDF (Term Frequency-Inverse Document Frequency) weighting scheme. This identifies which objects (e.g., a "helmet") are statistically significant for specific question categories (e.g., "Activity Recognition").
  2. Category Selector: This module predicts appropriate question categories for an image based on weighted object features, filtering out irrelevant options via a selection threshold.
  3. Question Generator: An LSTM-based decoder, initialized with category-specific representations, generates the final question. This "forces" the model to focus on image regions relevant to the chosen category.
Example of category-guided questions for positional reasoning and activity recognition.

Experimental Results

The model was evaluated on the TDIUC (Task-Driven Image Understanding Challenge) dataset, which contains over 1.6 million QA pairs across 12 categories. The results highlight a significant trade-off between diversity and ground-truth similarity.

Performance comparison of CGVQG against baseline models across BLEU and diversity metrics.

As shown in the table, CGVQG significantly outperforms baselines in diversity metrics. It achieved an mBLEU3 score of 8.48, whereas the GRNN-VGGNet scored 33.04 (note that for mBLEU metrics, a lower score indicates higher diversity and lower redundancy). However, the model's BLEU4 score (9.81) was lower than the GRNN-RCNN baseline (16.37). This suggests that while CGVQG is more creative and diverse, it is less likely to replicate the exact "obvious" questions found in human-labeled datasets, which typically focus on the most salient parts of an image.

Conclusion and Future Work

The CGVQG model demonstrates that anchoring questions in specific categories and using statistical weighting for visual objects can effectively combat the "boringness" of VQG. The primary challenge remains the "Saliency Gap"—the tendency of the model to focus on valid but non-obvious objects. Future research will focus on balancing category-driven diversity with human-perceived saliency to improve both the variety and the perceived quality of generated questions.

Summary updated

BibTeX
@inproceedings{liu2023category,
  author = {Hongfei Liu and Jiali Chen and Wenhao Fang and Jiayuan Xie and Yi Cai},
  title = {{Category-Guided Visual Question Generation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26991},
}

Loading related content...

Figure