Deep Bayesian Network for Visual Question Generation

Badri N. Patro, Vinod K. Kurmi, Sandeep Kumar, Vinay P. Namboodiri

Abstract

Generating natural questions from an image is a semantic task that requires using vision and language modalities to learn multimodal representations. Images can have multiple visual and language cues such as places, captions, and tags. In this paper, we propose a principled deep Bayesian learning framework that combines these cues to produce natural questions. We observe that with the addition of more cues and by minimizing uncertainty among cues, the Bayesian network becomes more confident. We propose a Minimizing Uncertainty of Mixture of Cues (MUMC) approach that minimizes uncertainty present in a mixture of cues experts for generating probabilistic questions. Our results show a remarkable similarity to natural questions as validated by a human study, and the proposed approach substantially improves over state-of-the-art benchmarks on quantitative metrics like BLEU-n, METEOR, ROUGE, and CIDEr.

Introduction

While Artificial Intelligence has become adept at answering questions (Visual Question Answering) or describing scenes (Image Captioning), the task of generating natural, engaging questions—Visual Question Generation (VQG)—remains a significant challenge. Most current VQG systems rely on simple encoder-decoder architectures that often produce robotic, generic, or irrelevant questions.

The core challenge is mimicking human curiosity, which is often triggered by specific context: sometimes the location matters most ("Is this a cowshed?"), while other times the action is key ("Are the horses running?"). To address this, a model must effectively extract and prioritize diverse cues—places, captions, and tags—while handling the inherent uncertainty when these cues conflict or when data is noisy.

Methodology: The MC-BMN Framework

The researchers at IIT Kanpur developed the Multi-Cue Bayesian Moderator Network (MC-BMN). The architecture is built on three main pillars: Representation Module, Bayesian Moderator, and Question Generator.

Detailed architecture of the MC-BMN framework showing the Encoder (Representation Module and Moderator) and the Decoder (Question Generator).

1. Bayesian Experts

The model uses three specialized "experts" to extract features:

  • Place Expert: Uses a Bayesian CNN (PlaceCNN) to identify scene categories.
  • Caption Expert: Uses a Bayesian LSTM to process semantic descriptions.
  • Tag Expert: Extracts nouns, verbs, and question types (Why, How, etc.) using POS tagging.

2. Bayesian Moderator (Mixture of Experts)

Instead of just concatenating these features, a moderator network weighs the "advice" from each expert. It uses a gating mechanism to decide which cue is most pertinent for a specific image, allowing for a principled fusion of information.

3. Minimizing Uncertainty of Mixture of Cues (MUMC)

This is the framework's core innovation. The authors model two types of uncertainty: Aleatoric (noise inherent in the data) and Epistemic (uncertainty in the model's parameters, captured via MC-Dropout). The MUMC framework uses a specialized loss function to minimize this uncertainty, forcing the model to become more "confident" as it integrates more cues.

Key Insights and Results

A central observation of this research is that "More Cues = More Confidence." As the network integrates more types of cues (moving from just image pixels to including places and tags), the Bayesian network's uncertainty decreases.

Feature variance plots showing that the Bayesian model's uncertainty (variance) decreases as more cues (Place, Tag, Caption) are added, compared to a non-Bayesian baseline.

Quantitative Performance

The MC-BMN model outperformed previous state-of-the-art (SOTA) methods across major quantitative metrics on the VQG-COCO dataset:

Method BLEU-1 METEOR ROUGE CIDEr
Natural [38] 19.2 19.7 - -
MDN [43] (Previous SOTA) 36.0 23.4 - -
MC-BMN (Ours) 40.7 22.6 41.9 49.7

Note: While METEOR is slightly lower than MDN in the summary table, the authors highlight a 20% improvement in BLEU scores over the original "Natural" baseline and a 13% improvement over single-cue baselines.

Beyond math, the authors conducted a "Naturalness" Turing Test. They asked 175 people to rate the questions, and the model's outputs were rated as "more or equally natural" compared to human ground-truth questions 61.8% of the time.

Conclusion and Future Work

The MC-BMN framework demonstrates that a principled Bayesian approach to cue fusion can generate questions that closely mimic human curiosity. However, the system remains sensitive to the quality of upstream models; using predicted captions instead of ground truth leads to a significant performance drop (BLEU-1 falls from 40.7 to 24.6). Future work includes testing on larger-scale datasets like Visual Genome and extending this uncertainty-minimization approach to other multimodal tasks like Image Captioning and Object Classification.

For more details, visit the project page: Deep Bayesian VQG.

Summary updated

Abstract

Generating natural questions from an image is a semantic task that requires using vision and language modalities to learn multimodal representations. Images can have multiple visual and language cues such as places, captions, and tags. In this paper, we propose a principled deep Bayesian learning framework that combines these cues to produce natural questions. We observe that with the addition of more cues and by minimizing uncertainty in the among cues, the Bayesian network becomes more confident. We propose a Minimizing Uncertainty of Mixture of Cues (MUMC), that minimizes uncertainty present in a mixture of cues experts for generating probabilistic questions. This is a Bayesian framework and the results show a remarkable similarity to natural questions as validated by a human study. We observe that with the addition of more cues and by minimizing uncertainty among the cues, the Bayesian framework becomes more confident. Ablation studies of our model indicate that a subset of cues is inferior at this task and hence the principled fusion of cues is preferred. Further, we observe that the proposed approach substantially improves over state-of-the-art benchmarks on the quantitative metrics (BLEU-n, METEOR, ROUGE, and CIDEr). Here we provide project link for Deep Bayesian VQG \url{https://delta-lab-iitk.github.io/BVQG/}
BibTeX
@inproceedings{patro2020deep,
  author = {Badri N. Patro and Vinod K. Kurmi and Sandeep Kumar and Vinay P. Namboodiri},
  title = {{Deep Bayesian Network for Visual Question Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093293},
}
Figure