Introduction
Visual Question Answering (VQA) requires an AI to understand an image deeply enough to answer natural language questions. Historically, attention mechanisms in VQA have focused on one of two scales: uniform image grids or specific object proposals. Image-grid level attention captures fine-grained details but often misses the broader semantic context of distinct objects. Conversely, object-level attention, typically using object detectors, captures semantics well but can overlook sub-object details or context outside the detected bounding boxes.
This paper introduces Reciprocal Attention Fusion (RAF), a novel architecture that addresses this "semantic gap" by simultaneously attending to both low-level image grids and high-level object proposals. By merging these reciprocal visual cues with linguistic features, RAF achieves state-of-the-art performance on major benchmarks.
Methodology
The RAF model utilizes a multi-branch architecture designed to exploit the reciprocal relationship between different visual scales.
Joint Feature Embedding
The model extracts three primary feature types:
- Language Features: Questions are processed via a Gated Recurrent Unit (GRU) initialized with Skip-thought vectors.
- Image-Level (Grid) Features: Extracted using a ResNeXt backbone to preserve spatial grid information.
- Object-Level Features: Extracted using a Faster R-CNN (with ResNet-101) to identify the top 36 object proposals.
Hierarchical Attention Fusion
To fuse these modalities efficiently, the authors employ Tucker Decomposition. This mathematical technique approximates high-dimensional bilinear interactions—where every question feature interacts with every image feature—using a compact "core tensor." This approach allows the model to capture complex multi-modal relationships while maintaining computational efficiency, requiring roughly half the complexity of previous bilinear pooling methods.
Reciprocal Co-attention
The model generates two sets of attention weights: one for the grid and one for the objects. These are fused into a single "co-attended" representation, which informs the final classifier to produce an answer.
Key Insights
- Complementary Visual Scales: Grid features and object features provide non-overlapping information. Grids capture "where" and fine textures, while object proposals capture "what" and semantic identity.
- Mathematical Efficiency: Tucker Fusion enables high-order feature interaction without the memory explosion typically associated with bilinear pooling.
- Top-Down Guidance: The question serves as a top-down signal, directing the model's attention to relevant parts of both the bottom-up grid and object data.
Experimental Results
The RAF model was evaluated on VQAv1 and VQAv2. It demonstrated significant improvements, particularly as a single-model entry.
On the VQAv2 dataset, RAF achieved a significant boost, reaching 67.4% on the Test-standard set, outperforming the previous state-of-the-art "Bottom-Up and Top-Down" model by 1.7%. On VQAv1, it improved single-model performance to 68.2%. Ablation studies confirmed that combining grid and object features (64.2% on the validation set) outperformed using either grid-only (58.0%) or object-only (63.9%) features alone.
Discussion and Future Work
Despite its success, the model faces challenges when attention maps become ambiguous, spreading focus too thin and leading to incorrect predictions. Additionally, the object-level branch is dependent on the pre-trained detector's quality; missed objects cannot be recovered by the attention mechanism. Future research could explore integrating these reciprocal features into iterative reasoning frameworks to better handle complex multi-step questions.
Summary updated