VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation

Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, Boqing Gong

Introduction

Visual Question Answering (VQA) has traditionally relied on implicit attention mechanisms—mathematical shortcuts that help a model focus on relevant image regions. However, these mechanisms are often "black boxes," learned only through the correctness of the final text answer. This lack of explicit guidance frequently leads to attention maps that do not align with human reasoning.

The VQS paper addresses this by bridging the gap between VQA and semantic segmentation. The authors argue that answering a question like "What is the man sitting on?" requires the model to first explicitly identify the "man" and the "bench." By providing pixel-level "visual answers," the researchers enable supervised attention and introduce a new task: Question-Focused Semantic Segmentation (QFSS).

The VQS Dataset

The core contribution of this work is the VQS dataset, which explicitly links instance segmentations from MS COCO to question-answer pairs from the VQA dataset. Human annotators selected existing COCO masks or drew new bounding boxes to provide the exact visual evidence needed to answer a specific question.

Examples from the VQS dataset showing questions paired with ground-truth segmentation masks that serve as visual answers.

The dataset contains nearly 100,000 questions across 37,868 images, providing a rich resource for grounding natural language in precise visual entities. Unlike previous datasets like VQA-HAT, which use "blurry" human-scratched maps, VQS provides sharp, object-aligned segmentation masks that offer a more rigorous target for model training.

Methodology

The researchers leverage the VQS data for two primary objectives:

1. Supervised Attention for VQA

Instead of allowing the model to learn attention latently, the VQS masks are used as explicit supervision. An attention module is trained to produce a weight map ($p_i$) that matches the human-selected regions. This supervised feature is then processed by a Multi-Layer Perceptron (MLP) to predict the text answer. This approach ensures the model "looks" at the right place for the right reasons.

2. Question-Focused Semantic Segmentation (QFSS)

In QFSS, the model's output is not text, but a segmentation mask. Two frameworks were proposed:

  • Mask Aggregation: This model uses object proposals (generated by SharpMask) and learns to calculate combination coefficients to merge the correct proposals based on the question features.
  • DeconvNet Baseline: An end-to-end encoder-decoder network that generates a mask from scratch using deconvolutional layers, conditioned on both the image and the question embedding.

Key Results and Insights

The research yielded several significant findings:

  • Performance Gains: Integrating supervised attention into a simple MLP model achieved 69.5% accuracy on the VQA multiple-choice task, outperforming many more complex co-attention models of that era.
  • Superiority of Object-Aligned Masks: Models trained on VQS masks significantly outperformed those using the "blurry" maps from VQA-HAT, demonstrating that precise object boundaries are crucial for effective supervision.
  • The "Oracle" Gap: A massive performance gap exists between current models (0.3256 IOU for Mask Aggregation) and an "Oracle" method (0.5709 IOU) which perfectly selects the best available COCO masks. This suggests that the primary challenge in VQA is not just mask quality, but the complex reasoning required to link language to visual evidence.

Conclusion

By providing explicit pixel-level targets, the VQS paper demonstrates that supervised attention is more effective than relying on latent variables. Moving toward "visual answers" through tasks like QFSS provides a more rigorous evaluation of a model's grounding and prevents it from relying solely on language priors. This work opens new avenues for research into more transparent and reason-based vision-language models.

Summary updated

BibTeX
@inproceedings{gan2017vqs,
  author = {Chuang Gan and Yandong Li and Haoxiang Li and Chen Sun and Boqing Gong},
  title = {{VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=Sybwe-M_ZB},
}
Figure