Abstract
Popularized as 'bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA). However, it is not clear whether the advantages of regions (e.g. better localization) are the key reasons for the success of bottom-up attention. In this paper, we revisit grid features for VQA, and find they can work surprisingly well - running more than an order of magnitude faster with the same accuracy (e.g. if pre-trained in a similar fashion). Through extensive experiments, we verify that this observation holds true across different VQA models (reporting a state-of-the-art accuracy on VQA 2.0 test-std, 72.71), datasets, and generalizes well to other tasks like image captioning. As grid features make the model design and training process much simpler, this enables us to train them end-to-end and also use a more flexible network design. We learn VQA models end-to-end, from pixels directly to answers, and show that strong performance is achievable without using any region annotations in pre-training. We hope our findings help further improve the scientific understanding and the practical application of VQA.
Introduction
Since the introduction of "Bottom-Up and Top-Down Attention" in 2018, the VQA community has almost exclusively used features extracted from bounding boxes (regions) identified by pre-trained object detectors. While highly accurate, this approach creates a massive computational bottleneck: the model must first run a complex detector to propose thousands of regions, select the best ones, and then compute features for each.
This research asks a fundamental question: Are regions actually superior to grids, or have we just been pre-training them better? The authors aim to prove that the format of the features (boxes vs. a standard grid) matters much less than the semantic information they contain and the resolution at which they are processed.
Methods
The researchers bridge the gap between regions and grids through several key technical steps. First, they align feature extraction by taking a standard Faster R-CNN detector and converting it back into a grid feature extractor. Instead of pooling features from specific bounding boxes, they utilize the output of the final convolutional layer ($C_5$).
To improve these grid features, they propose a pre-training modification where each region is represented by a single $1 \times 1$ vector rather than a $14 \times 14$ pool. This forces each grid cell to capture more localized, discriminative information. Because grid features are standard tensors, the authors can train the VQA model "end-to-end"—from raw pixels to the final answer—allowing the visual backbone to adapt specifically to the VQA task. Finally, to compensate for the lack of multi-scale "boxes," they add a Pyramid Pooling Module (PPM) on top of the grid features to aggregate visual information at different spatial resolutions.
Key Insights and Results
The core finding is that the success of region features was largely due to the Visual Genome dataset and high-resolution inputs, rather than the bounding boxes themselves. When grid features are given the same advantages, the performance gap vanishes.
Performance and Speed
- Inference Speed: The grid-based model is 48.3× faster than the region-based counterpart (18ms vs 889ms per image).
- VQA Accuracy: On VQA 2.0 test-std, the model achieved an accuracy of 72.71%, setting a new state-of-the-art for single-model performance at the time of publication.
- Localization: Attention mechanisms can "find" relevant parts of an image using grids just as effectively as they can with pre-defined boxes.
- End-to-End Benefit: Training end-to-end from pixels improved accuracy by up to 2.76% for models pre-trained on ImageNet, demonstrating that visual features can be optimized specifically for VQA.
Conclusion
This work demonstrates that the speed-accuracy tradeoff in VQA is a myth. By skipping the region proposal and selection steps, grid-based pipelines offer massive efficiency gains without sacrificing performance. While region-based modeling remains necessary for tasks requiring region outputs (like Referring Expression Grounding), grid features provide a simpler, more flexible, and faster alternative for VQA and image captioning. Future research should focus on better pre-training tasks and multi-scale grid aggregation rather than refining object detectors for these tasks.
Summary updated