Abstract
Visual attention mechanisms have proven to be integrally important constituent components of many modern deep neural architectures. They provide an efficient and effective way to utilize visual information selectively, which has shown to be especially valuable in multi-modal learning tasks. However, all prior attention frameworks lack the ability to explicitly model structural dependencies among attention variables, making it difficult to predict consistent attention masks. In this paper we develop a novel structured spatial attention mechanism which is end-to-end trainable and can be integrated with any feed-forward convolutional neural network. This proposed AttentionRNN layer explicitly enforces structure over the spatial attention variables by sequentially predicting attention values in the spatial mask in a bi-directional raster-scan and inverse raster-scan order. As a result, each attention value depends not only on local image or contextual information, but also on the previously predicted attention values. Our experiments show consistent quantitative and qualitative improvements on a variety of recognition tasks and datasets; including image categorization, question answering and image generation.
Introduction: The Need for Structured Attention
Most modern computer vision models utilize attention mechanisms to focus on relevant regions of an image. However, existing approaches generally suffer from a lack of spatial structure. Current mechanisms typically fall into two categories:
- Global Attention: Predicts all attention values simultaneously. While comprehensive, this is computationally expensive and prone to overfitting.
- Local Attention: Predicts each attention value independently based on a small local window. This often results in "noisy" masks with sharp discontinuities, where the model might attend to one pixel but ignore its immediate neighbor, even if they belong to the same object.
The authors argue that attention should be inherently structured. If a model identifies a portion of an object, that information should inform its decision to attend to the rest of that object.
Methodology: The AttentionRNN Layer
The core innovation is the AttentionRNN layer, a "plug-and-play" module designed to be integrated into existing architectures like ResNet. Instead of predicting attention values in a single pass, it treats the attention mask as a structured sequence.
Diagonal Traversal and Bi-directional Scanning
To capture spatial dependencies efficiently, the model employs a Diagonal BiLSTM (inspired by PixelRNN). It traverses the image feature map diagonally, ensuring that when predicting an attention value for a specific pixel, the model has already processed the context from adjacent pixels. By using two LSTMs—one moving from top-left to bottom-right and another from top-right to bottom-left—the layer ensures every pixel benefits from a global spatial context.
Efficiency via Block AttentionRNN (BRNN)
To address the computational intensity of sequential LSTMs on high-resolution images, the authors introduced a "Block" version. This variant downsamples the feature map into a coarse grid, computes structured attention, and then upsamples the resulting mask, maintaining structural integrity while significantly reducing overhead.
Experimental Evaluation
The AttentionRNN was evaluated across diverse benchmarks:
- Synthetic Attribute Prediction: On the noisy MBG dataset, AttentionRNN achieved 85.84% accuracy, significantly outperforming the local context baseline (79.00%).
- Image Classification: Integrating AttentionRNN into a ResNet18+CBAM architecture reduced the Top-1 error on CIFAR-100 from 25.07% to 24.18%.
- Visual Question Answering (VQA): The model provided a consistent boost over the Multimodal Compact Bilinear Pooling (MCB) baseline, reaching 57.58% accuracy on VQA 2.0.
- Image Generation: In attribute-conditioned image-to-image translation (CelebA), AttentionRNN produced smooth, uniform masks for tasks like hair color transformation, whereas standard models produced patchy, inconsistent attention.
Discussion and Key Insights
The primary takeaway is that spatial consistency in attention is functionally superior to independent pixel-wise prediction. By enforcing structure, the model becomes more robust to background noise and varying object scales. While the sequential nature of RNNs introduces a computational overhead (approximately 4.7x slower than simpler mechanisms), the qualitative and quantitative gains suggest that modeling spatial dependencies allows the network to "learn for the right reasons."
Conclusion
AttentionRNN introduces a novel way to model spatial dependencies in visual attention. By treating the attention mask as a structured sequence, it produces more coherent and effective masks across a wide range of tasks. Future work may involve exploring more efficient recurrent structures or Transformer-based architectures to achieve similar structural benefits with lower latency.
Summary updated