C3VQG: category consistent cyclic visual question generation

Shagun Uppal, Anish Madan, Sarthak Bhagat, Yi Yu, Rajiv Ratn Shah

Abstract

Visual Question Generation (VQG) is a critical task for developing autonomous agents capable of visual curiosity and information seeking. Traditional VQG systems rely heavily on supervised learning with expensive image-question-answer triplets. We present C3VQG (Category Consistent Cyclic Visual Question Generation), a novel framework that learns to generate diverse, relevant questions using only broad category labels (e.g., "color," "count") as weak supervision. By integrating a cyclic consistency loss with a structured latent space, C3VQG produces highly specific questions that significantly outperform previous weakly supervised methods in both automated metrics and human evaluation.

Introduction

Visual Question Generation (VQG) represents the "curiosity" side of computer vision: given an image, can a system ask a meaningful question about its content? Most existing VQG systems are trained using supervised learning, which requires large datasets of image-question-answer triplets. However, collecting ground-truth answers for every possible question is prohibitively expensive and tedious.

Figure 1: A complex visual scene where a VQG system might be asked to generate questions about activity, count, or spatial relationships.

Furthermore, without strong constraints, VQG models often default to "generic" questions (e.g., "What is in the image?") that fail to demonstrate true visual understanding. C3VQG addresses these challenges by creating a system that is weakly supervised (no ground-truth answers required), category-specific (controllable via attribute labels), and diverse (avoiding repetitive outputs).

Methodology

C3VQG utilizes a Variational Autoencoder (VAE) backbone enhanced by three key architectural innovations designed to enforce category relevance and feature disentanglement.

1. Category-Consistent Cyclic Loss

This mechanism establishes a feedback loop within the system. The model receives an image and a target category (e.g., "Activity") to generate a question. This generated question is then passed to a separate classifier that attempts to predict the original category. If the classifier cannot accurately identify the category from the question alone, the model is penalized. This ensures the generated text is semantically tied to the requested visual attribute.

2. Latent Space Clustering (Center Loss)

To improve the robustness of internal representations, the framework applies "Center Loss" to the latent space. This forces the latent features of questions belonging to the same category to cluster together in the vector space, enhancing the model's ability to distinguish between different types of visual cues and reducing category "blending."

Figure 2: The C3VQG architecture, highlighting the two-step process: Visual Question Generation (Step I) and Cycle Consistency Assurance (Step II).

3. Modified Hyper-prior

Rather than employing a standard Gaussian distribution for the latent space, C3VQG uses a learnable hyper-prior on the variance. This encourages the model to capture independent, disentangled visual features, which directly leads to more diverse and inventive question generation.

Experimental Results

The model was evaluated using the VQA (Visual Question Answering) dataset. During training, ground-truth answers were ignored to maintain the weakly supervised setting, focusing solely on images, questions, and category labels.

Quantitative Performance

C3VQG demonstrated substantial improvements over state-of-the-art weakly supervised models. In terms of the CIDEr metric, which measures similarity to human-written text, C3VQG achieved a score of 46.87, significantly outperforming the previous baseline of 35.89. The model also exhibited high "Inventiveness," generating a large percentage of unique questions not present in the training set.

Human Evaluation

In user studies, C3VQG achieved 97.8% relevance to the image and 60.5% relevance to the requested category. This represents a massive improvement over previous baselines, which averaged only 42.7% category relevance.

Conclusion

C3VQG demonstrates that ground-truth answers are not strictly necessary to train effective VQG models. By shifting the focus to category consistency and structured latent spaces, the model achieves superior performance with less supervision. Future work will explore extending this approach to conversational AI and image captioning, as well as enabling the model to autonomously discover new question categories in complex, crowded scenes.

Summary updated

BibTeX
@inproceedings{uppal2020c,
  author = {Shagun Uppal and Anish Madan and Sarthak Bhagat and Yi Yu and Rajiv Ratn Shah},
  title = {{C3VQG: category consistent cyclic visual question generation}},
  booktitle = {MMAsia},
  year = {2020},
  doi = {10.1145/3444685.3446302},
}
Figure