K-VQG: Knowledge-aware Visual Question Generation for Common-sense Acquisition

Kohei Uehara, Tatsuya Harada

Abstract

Visual Question Generation (VQG) is a task to generate questions from images. When humans ask questions about an image, their goal is often to acquire some new knowledge. However, existing studies on VQG have mainly addressed question generation from answers or question categories, overlooking the objectives of knowledge acquisition. To introduce a knowledge acquisition perspective into VQG, we constructed a novel knowledge-aware VQG dataset called K-VQG. This is the first large, humanly annotated dataset in which questions regarding images are tied to structured knowledge. We also developed a new VQG model that can encode and use knowledge as the target for a question. The experiment results show that our model outperforms existing models on the K-VQG dataset.

Introduction

Visual Question Generation (VQG) traditionally involves creating natural language questions based on visual input. However, conventional research often treats this as a "backward" task: models are provided with an image and a known answer, then tasked with reconstructing the question. This approach fails to capture the fundamental human motivation for questioning: the desire to fill gaps in one's knowledge.

In real-world scenarios, a system should ask questions to acquire specific, missing information. For instance, seeing an unfamiliar object in a kitchen should prompt a question that helps the system understand its function or identity. Existing VQG systems frequently produce generic questions (e.g., "What is on the table?") because they lack a specific knowledge-driven target. The K-VQG framework addresses this by shifting the focus toward knowledge acquisition, enabling models to generate questions that target specific common-sense triplets.

Examples of knowledge-aware visual question generation. The model uses a masked knowledge triplet and visual context to generate a targeted question.

The K-VQG Dataset

To support this new task, the authors introduced the K-VQG dataset, the first large-scale, human-annotated dataset linking visual questions to structured common-sense knowledge. Unlike the existing FVQA dataset, which is smaller and often contains questions that ignore image content, K-VQG requires the model to understand specific visual context to ask a valid question.

  • Data Sources: Images were sampled from Visual Genome; triplets were sourced from ConceptNet and ATOMIC20.
  • Scale: 16,098 questions across 13,648 images, involving 6,084 unique knowledge triplets.
  • Contextual Complexity: Humans naturally use visual landmarks (e.g., "the silver object next to the cat") to specify what they are asking about. K-VQG questions are longer and contain more "non-knowledge" words (3.35 per question vs. 0.99 in FVQA), indicating a higher reliance on visual context.

Methodology

The authors propose a "knowledge-aware" VQG task where the input consists of an image and a masked target knowledge triplet. Knowledge is represented as a triplet: <head, relation, tail> (e.g., <lion, IsA, feline>). To simulate a knowledge gap, one part is masked (e.g., <[MASK], IsA, feline>).

The core architecture is an encoder-decoder transformer:

  • Encoder (UNITER): A multi-modal transformer that fuses visual region features (extracted via Faster R-CNN) with text embeddings of the masked triplet.
  • Decoder (BART): An autoregressive transformer that takes the fused features and generates a natural language question designed to elicit the masked information.

Experimental Results

The K-VQG model was evaluated against several baselines, including Image-to-Question (I2Q) and models conditioned on answer categories (IC2Q).

  • Question Quality: The proposed model significantly outperformed baselines across standard metrics, achieving a CIDEr score of 131.04, compared to 31.01 for the best baseline (IC2Q).
  • Knowledge Consistency: Using a "Target Knowledge Parser," the researchers found their model achieved a Triplet-BLEU of 64.33, while baselines struggled to stay above 8.0.
  • Ablation Study: Removing the target knowledge input caused the CIDEr score to plummet to 33.92, proving that structured knowledge is the primary driver for generating specific, meaningful questions.

Discussion and Future Work

The authors acknowledge hurdles such as difficulty with small or hidden objects and occasional target ambiguity. However, this work paves the way for "proactive learning" systems—AI that identifies what it doesn't know and asks humans for clarification to build its own internal knowledge graph.

Summary updated

Abstract

Visual Question Generation (VQG) is a task to generate questions from images. When humans ask questions about an image, their goal is often to acquire some new knowledge. However, existing studies on VQG have mainly addressed question generation from answers or question categories, overlooking the objectives of knowledge acquisition. To introduce a knowledge acquisition perspective into VQG, we constructed a novel knowledge-aware VQG dataset called K-VQG. This is the first large, humanly annotated dataset in which questions regarding images are tied to structured knowledge. We also developed a new VQG model that can encode and use knowledge as the target for a question. The experiment results show that our model outperforms existing models on the K-VQG dataset.
BibTeX
@inproceedings{uehara2023k,
  author = {Kohei Uehara and Tatsuya Harada},
  title = {{K-VQG: Knowledge-aware Visual Question Generation for Common-sense Acquisition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=_gYRvuClho},
}

Loading related content...

Figure