Abstract
We propose the Recurrent Soft Attention Model (RSAM), which integrates visual attention from the original image into an LSTM memory cell through a down-sample network. The model recurrently transmits visual attention to the memory cells for glimpse mask generation, providing a natural way for attention integration and exploitation in general object recognition problems. Evaluated on the CIFAR-10 dataset, our experiments demonstrate that the down-sample network and feedback mechanism play a critical role in the network's overall effectiveness.
Introduction
Traditional Convolutional Neural Networks (CNNs) typically process entire images in a single pass, which differs significantly from the sequential, attentive nature of human vision. Humans recognize objects through a series of "glimpses," focusing on informative regions while filtering out irrelevant noise. While "hard" attention models (using reinforcement learning to crop images) have been explored, they are often difficult to train and limited to simple tasks. The Recurrent Soft Attention Model (RSAM) addresses this by using a differentiable soft attention mechanism to iteratively focus on relevant features in complex, real-world images.
The RSAM Architecture
The RSAM architecture is centered around a dual-LSTM structure that mimics the human glimpse mechanism. It consists of two primary components working in tandem:
- Context LSTM (C0): This cell maintains the global context and generates an attention mask ($M_t$). It receives input from a shallow Down-sample Network, which uses convolutional layers to provide a low-resolution overview of the input image.
- Glimpse LSTM (C1): This cell processes the "glimpse image"—the element-wise product of the original image and the attention mask. It extracts features from these focused regions to produce class probabilities.
A pivotal feature of RSAM is the Feedback Mechanism. The hidden state of the Glimpse LSTM (C1) is fed back into the Context LSTM (C0) for the subsequent time step. This loop allows the model to refine its attention mask based on information gathered in previous glimpses, leading to more stable and informed focus.
Experimental Results
The model was evaluated on the CIFAR-10 dataset, consisting of 60,000 32x32 color images across 10 classes. This dataset provides a more rigorous test for attention mechanisms than simpler digit-based datasets.
Ablation Study: Feedback and Down-sampling
To validate the architectural choices, the researchers compared the full RSAM against versions lacking the feedback loop or using standard fully connected layers instead of the convolutional down-sample network. As shown in the figure below, the combination of feedback and convolutional down-sampling significantly outperformed other configurations in terms of test accuracy and convergence speed.
Impact of Glimpse Count
The study also investigated the optimal number of glimpses. Accuracy improved as the number of glimpses increased from 2 to 4, reaching a peak Top-1 accuracy of 64.95%. Beyond four glimpses, performance plateaued, suggesting that four iterations are sufficient to capture the necessary information from 32x32 images. Qualitatively, the model progressively "fills in" the attention mask over successive glimpses, moving from sparse initial focus to a more comprehensive representation of the object.
Conclusion and Future Work
The RSAM demonstrates that a recurrent soft attention mechanism, supported by convolutional down-sampling and feedback, is effective for common object recognition. While the current implementation uses shallow networks to isolate the impact of the attention mechanism, future work could involve scaling the architecture. Integrating deeper glimpse networks, such as ResNets, or applying the model to higher-resolution datasets could further unlock the computational benefits of selective visual attention.
Summary updated