Abstract
Generating natural questions from an image is a semantic task that requires using vision and language modalities to learn multimodal representations. Images can have multiple visual and language cues such as places, captions, and tags. In this paper, we propose a principled deep Bayesian learning framework that combines these cues to produce natural questions. We observe that with the addition of more cues and by minimizing uncertainty among cues, the Bayesian network becomes more confident. We propose a Minimizing Uncertainty of Mixture of Cues (MUMC) approach that minimizes uncertainty present in a mixture of cues experts for generating probabilistic questions. Our results show a remarkable similarity to natural questions as validated by a human study, and the proposed approach substantially improves over state-of-the-art benchmarks on quantitative metrics like BLEU-n, METEOR, ROUGE, and CIDEr.
Introduction
While Artificial Intelligence has become adept at answering questions (Visual Question Answering) or describing scenes (Image Captioning), the task of generating natural, engaging questions—Visual Question Generation (VQG)—remains a significant challenge. Most current VQG systems rely on simple encoder-decoder architectures that often produce robotic, generic, or irrelevant questions.
The core challenge is mimicking human curiosity, which is often triggered by specific context: sometimes the location matters most ("Is this a cowshed?"), while other times the action is key ("Are the horses running?"). To address this, a model must effectively extract and prioritize diverse cues—places, captions, and tags—while handling the inherent uncertainty when these cues conflict or when data is noisy.
Methodology: The MC-BMN Framework
The researchers at IIT Kanpur developed the Multi-Cue Bayesian Moderator Network (MC-BMN). The architecture is built on three main pillars: Representation Module, Bayesian Moderator, and Question Generator.
1. Bayesian Experts
The model uses three specialized "experts" to extract features:
- Place Expert: Uses a Bayesian CNN (PlaceCNN) to identify scene categories.
- Caption Expert: Uses a Bayesian LSTM to process semantic descriptions.
- Tag Expert: Extracts nouns, verbs, and question types (Why, How, etc.) using POS tagging.
2. Bayesian Moderator (Mixture of Experts)
Instead of just concatenating these features, a moderator network weighs the "advice" from each expert. It uses a gating mechanism to decide which cue is most pertinent for a specific image, allowing for a principled fusion of information.
3. Minimizing Uncertainty of Mixture of Cues (MUMC)
This is the framework's core innovation. The authors model two types of uncertainty: Aleatoric (noise inherent in the data) and Epistemic (uncertainty in the model's parameters, captured via MC-Dropout). The MUMC framework uses a specialized loss function to minimize this uncertainty, forcing the model to become more "confident" as it integrates more cues.
Key Insights and Results
A central observation of this research is that "More Cues = More Confidence." As the network integrates more types of cues (moving from just image pixels to including places and tags), the Bayesian network's uncertainty decreases.
Quantitative Performance
The MC-BMN model outperformed previous state-of-the-art (SOTA) methods across major quantitative metrics on the VQG-COCO dataset:
| Method | BLEU-1 | METEOR | ROUGE | CIDEr |
|---|---|---|---|---|
| Natural [38] | 19.2 | 19.7 | - | - |
| MDN [43] (Previous SOTA) | 36.0 | 23.4 | - | - |
| MC-BMN (Ours) | 40.7 | 22.6 | 41.9 | 49.7 |
Note: While METEOR is slightly lower than MDN in the summary table, the authors highlight a 20% improvement in BLEU scores over the original "Natural" baseline and a 13% improvement over single-cue baselines.
Beyond math, the authors conducted a "Naturalness" Turing Test. They asked 175 people to rate the questions, and the model's outputs were rated as "more or equally natural" compared to human ground-truth questions 61.8% of the time.
Conclusion and Future Work
The MC-BMN framework demonstrates that a principled Bayesian approach to cue fusion can generate questions that closely mimic human curiosity. However, the system remains sensitive to the quality of upstream models; using predicted captions instead of ground truth leads to a significant performance drop (BLEU-1 falls from 40.7 to 24.6). Future work includes testing on larger-scale datasets like Visual Genome and extending this uncertainty-minimization approach to other multimodal tasks like Image Captioning and Object Classification.
For more details, visit the project page: Deep Bayesian VQG.
Summary updated