Visual Question Answering with Question Representation Update (QRU)

Ruiyu Li, Jiaya JiaNeurIPS 2016 · OpenReview

Introduction

The Question Representation Update (QRU) model moves beyond static question analysis in Visual Question Answering (VQA). By iteratively refining the question's internal representation through interactions with specific image regions, the model "learns" to make its internal query more specific to the visual context, achieving state-of-the-art performance on the COCO-QA and VQA benchmarks at the time of its release.

The Challenge of Static Representations

In standard VQA, models typically convert a question into a fixed mathematical vector and an image into a set of visual features, then "fuse" them to find an answer. The authors argue this is insufficient. A human doesn't just look at a question once; they look at the image, refine their understanding of what the question is asking based on what they see, and then look again.

Example of iterative question refinement

For example, if asked "What is sitting amongst things that have been abandoned?", a model first needs to identify "abandoned things" in the image. Only then can it refine its search to find what is "sitting" near them. Static models often lack this iterative "grounding," leading to vague representations that fail on complex or ambiguous queries.

The QRU Architecture

The QRU architecture introduces a dynamic reasoning process consisting of four main stages:

QRU Architecture Diagram
  1. Image Understanding: Instead of looking at the whole image as a single block, the model uses Edge Boxes to propose 19 candidate object regions plus one global image view. Features are extracted using a pre-trained VGG-19 network. Crucially, the authors append 8D spatial coordinates to these features to help the model understand where objects are located.
  2. Question Encoding: Questions are processed through a Gated Recurrent Unit (GRU) initialized with skip-thought vectors (a technique for general sentence embedding). This provides a strong linguistic starting point.
  3. The Reasoning Layer (The Core Innovation): This layer performs an iterative update. The current question representation interacts with each image region via a Multi-Layer Perceptron (MLP), utilizing element-wise multiplication between the question and visual features. The model calculates attention weights for each region based on how well they relate to the current question. It then creates a new, "updated" question representation by taking a weighted average of these interactions.
  4. Answering Layer: The final updated representation is passed to a softmax classifier to choose the most likely answer from a pre-defined set (the top 1,000 most frequent answers).

Key Insights and Performance

The model was evaluated on two major benchmarks: COCO-QA and VQA (v1). On COCO-QA, the two-layer QRU achieved 62.50% accuracy, outperforming the Stacked Attention Network (SAN) by nearly 1%. On the VQA test-dev set, it reached 60.72% for Open-Ended and 65.43% for Multiple-Choice tasks.

Several key insights emerged from the model's design:

  • Iterative Refinement: The core "aha moment" is that the question representation shouldn't be static. By allowing the question vector to change after "seeing" image regions, the model effectively performs a multi-step reasoning process.
  • Spatial Awareness is Vital: Ablation studies showed that removing spatial coordinates led to a significant drop in accuracy, especially for "Object" type questions. Knowing where an object is helps the model reason about relationships (e.g., "What is next to the building?").
  • Proposals vs. Grids: Unlike contemporary models that used a coarse grid, QRU uses object proposals. This allows the model to focus on meaningful entities rather than arbitrary squares of pixels.
  • Language Evolution: Qualitatively, as the model goes through reasoning layers, the "internal" question becomes more relevant to the image. An initial vague query becomes more similar to specific descriptions of the objects actually present in the scene.

Limitations and Future Work

While highly effective, the QRU model has limitations. It struggled with "Number" questions because it relies on object proposals; if the proposal generator misses an object or creates overlapping boxes for the same object, the counting logic fails. Additionally, researchers found diminishing returns with deeper networks—adding a third reasoning layer provided no further improvement. Future work aims to improve the fine-grained relationship between specific words in the question and specific regions in the image.

Summary updated

Abstract

Our method aims at reasoning over natural language questions and visual images. Given a natural language question about an image, our model updates the question representation iteratively by selecting image regions relevant to the query and learns to give the correct answer. Our model contains several reasoning layers, exploiting complex visual relations in the visual question answering (VQA) task. The proposed network is end-to-end trainable through back-propagation, where its weights are initialized using pre-trained convolutional neural network (CNN) and gated recurrent unit (GRU). Our method is evaluated on challenging datasets of COCO-QA and VQA and yields state-of-the-art performance.
BibTeX
@inproceedings{li2016visual,
  author = {Ruiyu Li and Jiaya Jia},
  title = {{Visual Question Answering with Question Representation Update (QRU)}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2016},
  url = {https://openreview.net/forum?id=B1W3D_b_Zr},
}
Figure