Introduction
The Question Representation Update (QRU) model moves beyond static question analysis in Visual Question Answering (VQA). By iteratively refining the question's internal representation through interactions with specific image regions, the model "learns" to make its internal query more specific to the visual context, achieving state-of-the-art performance on the COCO-QA and VQA benchmarks at the time of its release.
The Challenge of Static Representations
In standard VQA, models typically convert a question into a fixed mathematical vector and an image into a set of visual features, then "fuse" them to find an answer. The authors argue this is insufficient. A human doesn't just look at a question once; they look at the image, refine their understanding of what the question is asking based on what they see, and then look again.
For example, if asked "What is sitting amongst things that have been abandoned?", a model first needs to identify "abandoned things" in the image. Only then can it refine its search to find what is "sitting" near them. Static models often lack this iterative "grounding," leading to vague representations that fail on complex or ambiguous queries.
The QRU Architecture
The QRU architecture introduces a dynamic reasoning process consisting of four main stages:
- Image Understanding: Instead of looking at the whole image as a single block, the model uses Edge Boxes to propose 19 candidate object regions plus one global image view. Features are extracted using a pre-trained VGG-19 network. Crucially, the authors append 8D spatial coordinates to these features to help the model understand where objects are located.
- Question Encoding: Questions are processed through a Gated Recurrent Unit (GRU) initialized with skip-thought vectors (a technique for general sentence embedding). This provides a strong linguistic starting point.
- The Reasoning Layer (The Core Innovation): This layer performs an iterative update. The current question representation interacts with each image region via a Multi-Layer Perceptron (MLP), utilizing element-wise multiplication between the question and visual features. The model calculates attention weights for each region based on how well they relate to the current question. It then creates a new, "updated" question representation by taking a weighted average of these interactions.
- Answering Layer: The final updated representation is passed to a softmax classifier to choose the most likely answer from a pre-defined set (the top 1,000 most frequent answers).
Key Insights and Performance
The model was evaluated on two major benchmarks: COCO-QA and VQA (v1). On COCO-QA, the two-layer QRU achieved 62.50% accuracy, outperforming the Stacked Attention Network (SAN) by nearly 1%. On the VQA test-dev set, it reached 60.72% for Open-Ended and 65.43% for Multiple-Choice tasks.
Several key insights emerged from the model's design:
- Iterative Refinement: The core "aha moment" is that the question representation shouldn't be static. By allowing the question vector to change after "seeing" image regions, the model effectively performs a multi-step reasoning process.
- Spatial Awareness is Vital: Ablation studies showed that removing spatial coordinates led to a significant drop in accuracy, especially for "Object" type questions. Knowing where an object is helps the model reason about relationships (e.g., "What is next to the building?").
- Proposals vs. Grids: Unlike contemporary models that used a coarse grid, QRU uses object proposals. This allows the model to focus on meaningful entities rather than arbitrary squares of pixels.
- Language Evolution: Qualitatively, as the model goes through reasoning layers, the "internal" question becomes more relevant to the image. An initial vague query becomes more similar to specific descriptions of the objects actually present in the scene.
Limitations and Future Work
While highly effective, the QRU model has limitations. It struggled with "Number" questions because it relies on object proposals; if the proposal generator misses an object or creates overlapping boxes for the same object, the counting logic fails. Additionally, researchers found diminishing returns with deeper networks—adding a third reasoning layer provided no further improvement. Future work aims to improve the fine-grained relationship between specific words in the question and specific regions in the image.
Summary updated