Deep Bayesian Network for Visual Question Generation

Badri N. Patro, Vinod K. Kurmi, Sandeep Kumar, Vinay P. Namboodiri

Introduction

While Artificial Intelligence has made significant strides in Visual Question Answering (VQA), the inverse task—Visual Question Generation (VQG)—remains a formidable challenge. Humans naturally generate questions driven by curiosity and context, yet most AI systems produce generic, robotic queries like "What is in the image?" This lack of specificity stems from the inherent ambiguity of visual data; a single scene can be interpreted through various lenses, such as location, action, or specific objects.

Researchers from IIT Kanpur have addressed this by developing a deep Bayesian framework that mimics human-like curiosity. Their model, the Multi-Cue Bayesian Moderator Network (MC-BMN), explicitly models uncertainty and fuses multiple visual cues to generate natural, engaging questions.

The Multi-Cue Bayesian Moderator Network (MC-BMN)

The core of the proposed system is a "Mixture of Experts" approach that processes three distinct types of contextual "advice":

  1. Place Expert: Identifies the scene category (e.g., a cowshed or kitchen).
  2. Caption Expert: Generates descriptive text to provide narrative context.
  3. Tag Expert: Extracts specific nouns, verbs, and question-type tags.
Conceptual overview of the Moderator network weighing advice from Place, Caption, and Tag experts to synthesize a final question.

To manage these inputs, a Moderator Network acts as a gatekeeper. It dynamically weighs the relevance of each expert based on the specific image. For instance, in a wide-angle shot of a landscape, the "Place" cue might be prioritized, whereas a close-up of an activity might rely more heavily on "Tags."

Bayesian Architecture and Uncertainty Modeling

Unlike standard deterministic models, the MC-BMN utilizes a Bayesian Representation Module. By employing Bayesian CNNs and LSTMs (implemented via Monte Carlo Dropout), the model can estimate epistemic uncertainty (uncertainty in model weights).

Detailed architecture of the MC-BMN, showing the Encoder (Representation Module and Moderator) and the Decoder (Question Generator).

The researchers introduced a novel loss function called Minimizing Uncertainty of Mixture of Cues (MUMC). This function forces the network to reduce both aleatoric uncertainty (noise in the data) and epistemic uncertainty, leading to a more "confident" and human-like fusion of information. By treating question generation as a probabilistic task, the model samples from a distribution of likely human responses rather than settling for a safe, generic average.

Key Results and Human Evaluation

The MC-BMN was evaluated on the VQG-COCO benchmark, where it achieved a BLEU-1 score of 40.7, significantly outperforming the previous state-of-the-art (36.0). Beyond quantitative metrics, the model underwent a "Turing Test" style human study with 175 participants.

In this study, humans were asked to compare AI-generated questions against ground-truth human annotations. Remarkably, 61.8% of the time, the AI-generated questions were rated as being as natural as, or more natural than, those written by humans. This suggests that the Bayesian fusion of diverse cues successfully captures the nuance of human curiosity.

Conclusion and Future Directions

The success of the MC-BMN highlights the importance of uncertainty modeling in multimodal tasks. While the model currently relies on pre-trained experts, future work could involve end-to-end training on larger, more complex datasets like Visual Genome. The authors suggest that this Bayesian moderator approach could be extended to other domains, such as image captioning or interactive dialogue systems, where resolving visual ambiguity is critical.

Summary updated

Abstract

Generating natural questions from an image is a semantic task that requires using vision and language modalities to learn multimodal representations. Images can have multiple visual and language cues such as places, captions, and tags. In this paper, we propose a principled deep Bayesian learning framework that combines these cues to produce natural questions. We observe that with the addition of more cues and by minimizing uncertainty in the among cues, the Bayesian network becomes more confident. We propose a Minimizing Uncertainty of Mixture of Cues (MUMC), that minimizes uncertainty present in a mixture of cues experts for generating probabilistic questions. This is a Bayesian framework and the results show a remarkable similarity to natural questions as validated by a human study. We observe that with the addition of more cues and by minimizing uncertainty among the cues, the Bayesian framework becomes more confident. Ablation studies of our model indicate that a subset of cues is inferior at this task and hence the principled fusion of cues is preferred. Further, we observe that the proposed approach substantially improves over state-of-the-art benchmarks on the quantitative metrics (BLEU-n, METEOR, ROUGE, and CIDEr). Here we provide project link for Deep Bayesian VQG \url{https://delta-lab-iitk.github.io/BVQG/}
BibTeX
@inproceedings{patro2020deep,
  author = {Badri N. Patro and Vinod K. Kurmi and Sandeep Kumar and Vinay P. Namboodiri},
  title = {{Deep Bayesian Network for Visual Question Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093293},
}
Figure