TL;DR
This paper uses a deep convolutional neural network (DCNN) to model how spatial attention improves visual recognition. They found that suppressing irrelevant information (SUAR) is more effective than enhancing relevant information (EAR) when attention is applied to a single layer, especially in cluttered scenes. However, when attention is applied across multiple layers, EAR becomes as effective as SUAR, suggesting different attentional strategies are employed at different stages of visual processing.
What Problem Does This Paper Address?
Understanding how attention works in the brain is crucial for explaining how we perceive the world, especially in complex and visually noisy environments. While we know attention modulates neural activity, the precise mechanisms by which this modulation translates into improved behavioral performance on challenging tasks remain unclear. Specifically, researchers are interested in how attention might enhance processing in attended regions and/or suppress processing in unattended regions, and how these mechanisms might operate across the hierarchical structure of the visual system. This paper aims to bridge this gap by using a DCNN, a computational model inspired by the visual cortex, to investigate these attentional mechanisms and their impact on object recognition.
Methods
The authors employed a VGG-16 DCNN, a well-established architecture for image recognition, as their model of the visual system. They simulated two primary attentional mechanisms:
- Enhancement in Attended Region (EAR): This involves increasing the neural activity in the attended part of the visual scene.
- Suppression in Unattended Region (SUAR): This involves decreasing the neural activity in the unattended parts of the visual scene.
These mechanisms were implemented by directly modulating the slopes of the activation functions of artificial neurons within the DCNN. They applied these modulations in three ways: EAR only, SUAR only, or a combination of both (EAR+SUAR). Crucially, they investigated the effects of applying these modulations to:
- A single convolutional layer: This allowed them to test the impact of focal attention at a specific processing stage.
- Multiple successive convolutional layers (stacks): This allowed them to explore how attention might operate across different levels of the visual hierarchy.
The network's performance was evaluated on object categorization tasks using cluttered images with varying contrast levels.
Datasets
- ImageNet: The VGG-16 network was pre-trained on the standard ImageNet dataset for object recognition.
- ImageNet validation set: Used for training the binary classifiers within the VGG-16 model.
- Custom composite images: Created by scaling down ImageNet images and arranging a target image with distracters in a two-by-two grid. These images were manipulated for contrast levels to create challenging recognition tasks.
Key Insights and Ideas
- SUAR shines in single-layer, cluttered scenes: When attention was applied to a single layer of the DCNN, suppressing irrelevant information (SUAR) was significantly more effective than enhancing relevant information (EAR) for improving object recognition, especially when high-contrast distractors were present. This suggests that reducing noise is a primary benefit of focal attention at early processing stages.
- EAR struggles with low contrast alone: While SUAR could recover performance on low-contrast targets, EAR alone failed to do so, highlighting its limitations when the target signal is weak.
- Multi-layer EAR becomes a powerhouse: When EAR was applied diffusely across multiple successive layers of the DCNN, it achieved performance levels comparable to SUAR, even across varying target contrasts. This suggests that the cumulative effect of enhancement across the hierarchy can be a powerful attentional strategy.
- Two distinct attentional strategies: The study reveals two functionally equivalent but mechanistically different attentional strategies: focal suppression at later stages (SUAR) and diffuse enhancement across multiple stages (EAR). This provides a computational framework for understanding how attention might operate differently across the visual processing hierarchy.
Key Results
- Single-layer SUAR outperforms EAR: On full-contrast composite images, SUAR applied to the last convolutional layer boosted performance more than EAR. SUAR was also effective in recovering performance on low-contrast targets, where EAR failed.
- Multi-layer EAR matches SUAR: When attention was applied to stacks of successive layers, both EAR and SUAR (and their combination) were effective in recovering performance degraded by low contrast and clutter. In this multi-layer scenario, EAR exhibited comparable or even greater behavioral benefits than SUAR.
- SUAR is sensitive to clutter: The benefit of SUAR was significantly reduced on control images with no distractors, indicating its effectiveness is tied to reducing interference from irrelevant stimuli.
- EAR benefits from accumulation: The effectiveness of EAR in multi-layer applications is attributed to the accumulation of its effects across layers, leading to a substantial increase in signal power.
Limitations and Future Work
- Simplified DCNN: The authors acknowledge that their DCNN model lacks some neurobiological details, such as recurrent and feedback pathways, diverse cell types, and noisy neural activity. Future work could incorporate these elements to create more biologically realistic models.
- Deterministic neurons: The artificial neurons in their VGG-16 model are deterministic, meaning they did not explore attentional correlates like changes in trial-to-trial variability or noise correlations. Future studies could incorporate noisy spiking neuron models.
- Biological relevance of layer correspondence: The assumption that convolutional layers correspond to specific visual areas is still debated. Further research is needed to clarify the extent of biological neural circuits represented by convolutional layers.
- Future directions: The authors suggest future investigations should explore the functional distinction between enhancement and suppression in more biologically realistic models. They also propose testing their predictions about attentional suppression in higher visual areas and the cumulative effects of enhancement across the hierarchy in biological systems through future recordings.
Summary updated