Abstract
Visual Question Generation (VQG) is a task to generate questions from images. When humans ask questions about an image, their goal is often to acquire some new knowledge. However, existing studies on VQG have mainly addressed question generation from answers or question categories, overlooking the objectives of knowledge acquisition. To introduce a knowledge acquisition perspective into VQG, we constructed a novel knowledge-aware VQG dataset called K-VQG. This is the first large, humanly annotated dataset in which questions regarding images are tied to structured knowledge. We also developed a new VQG model that can encode and use knowledge as the target for a question. The experiment results show that our model outperforms existing models on the K-VQG dataset.
Introduction
Visual Question Generation (VQG) typically involves training models to generate relevant questions based on visual input. However, traditional VQG research often operates "backwards" by providing the model with the answer as an input. This contradicts the real-world utility of questioning, where agents (like humans or AI) ask questions to fill gaps in their knowledge. If a machine already knows the answer, the question is performative rather than inquisitive.
The core challenge in moving away from answer-conditioning is providing a precise control signal. Without an answer to guide it, how does a model know which specific aspect of an image to query? K-VQG addresses this by using structured common-sense knowledge as the guiding signal for question generation.
The K-VQG Dataset
To support this new paradigm, the researchers introduced the K-VQG dataset, the first large-scale, human-annotated VQG dataset tied to structured knowledge. It consists of 16,098 questions across 13,648 images sampled from Visual Genome.
Knowledge triplets were sourced from ConceptNet and ATOMIC20. Crowd workers on Amazon Mechanical Turk (AMT) were tasked with selecting appropriate triplets for specific image regions and writing natural language questions that target the missing component of those triplets. Unlike previous datasets, K-VQG encourages complex spatial reasoning and descriptive context, avoiding generic phrases like "in the image."
Methodology: Knowledge as a Control Signal
The proposed model architecture utilizes a transformer-based encoder-decoder framework. The encoder (UNITER) fuses visual region features (extracted via Faster R-CNN) with Target Knowledge Embeddings.
Knowledge is represented as structured triplets: <head, relation, tail>. To simulate the need for acquisition, the model is provided with a "masked" triplet where one element is hidden (e.g., <coaster, used for, [MASK]>). A BART-based decoder then generates a question designed to elicit the masked information. This approach allows for fine-grained control, enabling the model to ask about specific properties like materials or functions rather than just object identification.
Experimental Results
The K-VQG model was evaluated against several baselines, including I2Q and IM-VQG.
- Question Quality: The K-VQG model achieved a CIDEr score of 131.04, significantly outperforming the best baseline (31.01).
- Knowledge Consistency: Using a "Target Knowledge Parser," the model reached a 64.33 Triplet-BLEU score, demonstrating its ability to generate questions that accurately target the intended knowledge.
- Ablation Studies: Performance dropped sharply when either visual or knowledge inputs were removed, confirming that the model successfully grounds abstract knowledge in the visual scene.
Conclusion
K-VQG represents a shift toward "learning-by-asking." By grounding question generation in structured common-sense knowledge, this framework moves AI closer to autonomous agents capable of exploring environments and actively acquiring new information from human interlocutors. Future work will focus on improving handling of small or obscured objects and reducing ambiguity in entity targeting.
Summary updated