Abstract
Answer grounding in Visual Question Answering (VQA) aims to identify the specific visual evidence supporting a model's prediction. This paper introduces the Sentence Attention Block (SAB), a modular architectural component that adapts Squeeze-and-Excitation principles for multimodal data. By re-calibrating multi-scale visual features using joint question-answer embeddings, the proposed architecture achieves state-of-the-art performance on grounding benchmarks like VizWiz and TextVQA-X while remaining significantly more efficient than large-scale transformer models.
Introduction
While Visual Question Answering (VQA) has seen rapid progress, many high-performing models operate as "black boxes," providing correct answers without explicit visual justification. Answer Grounding addresses this by requiring models to highlight relevant image regions. Current research in this area faces three primary challenges: incompatibility with standard pre-trained backbones, excessive architectural complexity that hinders deployment, and a lack of foundational design principles.
To address these issues, we propose the Sentence Attention Block (SAB). Inspired by the Squeeze-and-Excitation (SE) block—a standard in image classification—the SAB is designed to be a lightweight, modular plug-in that can be integrated into existing vision-language pipelines to improve interpretability and grounding accuracy.
Methodology
System Overview
The overall architecture follows a multi-scale approach to capture both global context and fine-grained details. As shown in the system diagram, the model processes an input image through a CNN backbone to extract features at multiple resolutions (typically 1/32, 1/16, 1/8, and 1/4 of the original size). Simultaneously, the question and answer are processed through sentence embedding layers and concatenated to form a joint multimodal query.
These multi-scale features are then passed through individual Sentence Attention Blocks. The outputs of these blocks are upsampled and combined to produce the final answer grounding mask.
The Sentence Attention Block (SAB)
The core innovation of this work is the SAB, which re-calibrates image feature maps based on textual context. Unlike standard self-attention, the SAB employs a cross-attention mechanism where the sentence embedding serves as the external guide to filter visual channels.
Key design choices in the SAB include:
- Inverted Bottleneck: While standard SE blocks compress data to save computation, the SAB expands the embedding space by a factor of two (4E expansion from a 2E input). This allows the model to capture more complex, high-dimensional relationships between text and vision.
- Softmax Selection: A Softmax layer replaces the traditional Sigmoid activation. This forces the model to be more decisive, effectively selecting the most relevant visual channels for the given question-answer pair.
- Layer Normalization: Integrated LayerNorm ensures stable training across different scales and modalities.
Training Objectives
In addition to standard cross-entropy loss, the model utilizes Region Mutual Information (RMI) loss. RMI loss accounts for the relationships between neighboring pixels, encouraging the model to produce cohesive and clean segmentation masks rather than fragmented heatmaps.
Experimental Results
Benchmarks and Performance
The SAB-based model was evaluated on four major datasets: TextVQA-X, VQS, VQA-X, and VizWiz-VQA-Grounding. The model achieved 1st place on the VizWiz-VQA-Grounding 2022 leaderboard with a Mean IoU of 72.4. It also established new state-of-the-art results on TextVQA-X (29.0 Mean IoU) and VQS (36.8 Mean IoU), outperforming more complex models like MAC-Caps.
Efficiency and Scalability
A major advantage of the SAB is its efficiency. By leveraging powerful pre-trained backbones like EfficientNet-B7 and MiniLMv2, the model achieves superior results with only 88M parameters—a fraction of the size of competitors like VinVL (406M). Furthermore, the model boasts an inference time of just 49ms, making it suitable for real-time applications and edge devices.
Conclusion
The Sentence Attention Block demonstrates that effective answer grounding does not require massive, convoluted cross-encoders. By re-engineering proven architectural blocks for multimodal tasks, we can achieve state-of-the-art performance with high efficiency. Future work will explore the sensitivity of the SAB to different backbones and investigate "soft" grounding techniques to represent varying levels of model confidence.
Summary updated