Abstract
Visual Question Generation (VQG) is a task to generate questions from images. When humans ask questions about an image, their goal is often to acquire some new knowledge. However, existing studies on VQG have mainly addressed question generation from answers or question categories, overlooking the objectives of knowledge acquisition. To introduce a knowledge acquisition perspective into VQG, we constructed a novel knowledge-aware VQG dataset called K-VQG. This is the first large, humanly annotated dataset in which questions regarding images are tied to structured knowledge. We also developed a new VQG model that can encode and use knowledge as the target for a question. The experiment results show that our model outperforms existing models on the K-VQG dataset.
Introduction
Visual Question Generation (VQG) traditionally involves creating natural language questions based on visual input. However, conventional research often treats this as a "backward" task: models are provided with an image and a known answer, then tasked with reconstructing the question. This approach fails to capture the fundamental human motivation for questioning: the desire to fill gaps in one's knowledge.
In real-world scenarios, a system should ask questions to acquire specific, missing information. For instance, seeing an unfamiliar object in a kitchen should prompt a question that helps the system understand its function or identity. Existing VQG systems frequently produce generic questions (e.g., "What is on the table?") because they lack a specific knowledge-driven target. The K-VQG framework addresses this by shifting the focus toward knowledge acquisition, enabling models to generate questions that target specific common-sense triplets.
The K-VQG Dataset
To support this new task, the authors introduced the K-VQG dataset, the first large-scale, human-annotated dataset linking visual questions to structured common-sense knowledge. Unlike the existing FVQA dataset, which is smaller and often contains questions that ignore image content, K-VQG requires the model to understand specific visual context to ask a valid question.
- Data Sources: Images were sampled from Visual Genome; triplets were sourced from ConceptNet and ATOMIC20.
- Scale: 16,098 questions across 13,648 images, involving 6,084 unique knowledge triplets.
- Contextual Complexity: Humans naturally use visual landmarks (e.g., "the silver object next to the cat") to specify what they are asking about. K-VQG questions are longer and contain more "non-knowledge" words (3.35 per question vs. 0.99 in FVQA), indicating a higher reliance on visual context.
Methodology
The authors propose a "knowledge-aware" VQG task where the input consists of an image and a masked target knowledge triplet. Knowledge is represented as a triplet: <head, relation, tail> (e.g., <lion, IsA, feline>). To simulate a knowledge gap, one part is masked (e.g., <[MASK], IsA, feline>).
The core architecture is an encoder-decoder transformer:
- Encoder (UNITER): A multi-modal transformer that fuses visual region features (extracted via Faster R-CNN) with text embeddings of the masked triplet.
- Decoder (BART): An autoregressive transformer that takes the fused features and generates a natural language question designed to elicit the masked information.
Experimental Results
The K-VQG model was evaluated against several baselines, including Image-to-Question (I2Q) and models conditioned on answer categories (IC2Q).
- Question Quality: The proposed model significantly outperformed baselines across standard metrics, achieving a CIDEr score of 131.04, compared to 31.01 for the best baseline (IC2Q).
- Knowledge Consistency: Using a "Target Knowledge Parser," the researchers found their model achieved a Triplet-BLEU of 64.33, while baselines struggled to stay above 8.0.
- Ablation Study: Removing the target knowledge input caused the CIDEr score to plummet to 33.92, proving that structured knowledge is the primary driver for generating specific, meaningful questions.
Discussion and Future Work
The authors acknowledge hurdles such as difficulty with small or hidden objects and occasional target ambiguity. However, this work paves the way for "proactive learning" systems—AI that identifies what it doesn't know and asks humans for clarification to build its own internal knowledge graph.
Summary updated