TL;DR
This paper introduces OverLoCK, a novel ConvNet architecture that mimics human vision's top-down attention mechanism. By decomposing the network into "overview" and "look closely" stages and employing a new "context-mixing dynamic convolution" (ContMix), OverLoCK achieves state-of-the-art performance on image classification, object detection, and semantic segmentation with improved efficiency.
What Problem Does This Paper Address?
Modern Convolutional Neural Networks (CNNs) typically process images through a hierarchical structure that progressively downsamples features, expanding receptive fields. While effective, this approach neglects a key principle of human vision: the brain first forms a rough "overview" of a scene to identify salient cues, then performs a more detailed "look closely" analysis. Existing CNNs struggle to explicitly incorporate this top-down attention, leading to limitations in accurately localizing objects and capturing fine-grained details, especially at higher resolutions. This paper aims to bridge this gap by designing a ConvNet architecture that explicitly leverages top-down contextual guidance.
Methods
The core of OverLoCK lies in its Deep-stage Decomposition Strategy (DDS), which splits the network into three synergistic sub-networks:
- Base-Net: Encodes low- and mid-level features, similar to traditional CNN backbones.
- Overview-Net: A lightweight network that rapidly generates a semantically meaningful but low-quality "overview" feature map from the Base-Net's output. This serves as a "context prior."
- Focus-Net: A more powerful network that refines features and expands receptive fields, guided by the context prior from Overview-Net.
To effectively utilize this top-down guidance, OverLoCK introduces a novel Context-Mixing Dynamic Convolution (ContMix). ContMix dynamically generates convolution kernels for each token by considering its affinity with regional centers in the context prior. This allows the kernels to capture long-range dependencies while preserving the inherent local inductive biases of convolutions, a capability lacking in previous dynamic convolution methods. The architecture is further enhanced by a gated mechanism in the Focus-Net's Gated Dynamic Spatial Aggregator (GDSA) to filter contextual noise.
Datasets
The paper uses the following datasets:
- ImageNet-1K: For pre-training and evaluating image classification performance.
- COCO 2017: For evaluating object detection and instance segmentation.
- ADE20K: For evaluating semantic segmentation.
- ImageNet-V2, ImageNet-A, ImageNet-R, ImageNet-Sketch: For robustness evaluation.
Key Insights and Ideas
- Biomimicry for CNNs: The paper demonstrates that mimicking human visual attention mechanisms (overview-first, look-closely-next) can lead to significant improvements in CNN performance and efficiency.
- Decomposition for Context: Splitting the network into distinct "overview" and "focus" stages allows for explicit top-down contextual guidance, which is crucial for accurate perception.
- Dynamic Kernels for Long-Range Dependencies: ContMix effectively addresses the limitation of fixed convolutional kernels by generating context-aware, dynamic kernels that can capture long-range spatial relationships without sacrificing local inductive biases.
- ConvNets Can Compete with Transformers: OverLoCK shows that pure ConvNet architectures, when designed with advanced mechanisms like DDS and ContMix, can achieve performance competitive with, and often superior to, Transformer and Mamba-based models, especially in terms of efficiency.
Key Results
- Image Classification: OverLoCK-T achieves 84.2% Top-1 accuracy on ImageNet-1K, outperforming ConvNeXt-B with significantly fewer FLOPs and parameters (approximately one-third). OverLoCK-B reaches 85.1% Top-1 accuracy.
- Object Detection: OverLoCK-S surpasses MogaNet-B by 1% in AP$^b$ on the COCO dataset using Cascade Mask R-CNN.
- Semantic Segmentation: OverLoCK-T improves UniRepLKNet-T by 1.7% in mIoU on ADE20K using UperNet.
- Efficiency: OverLoCK demonstrates an excellent speed-accuracy trade-off, often achieving higher accuracy at faster inference speeds compared to other models. For example, OverLoCK-S surpasses MogaNet-B by over 100 imgs/s.
- Robustness: OverLoCK shows strong performance on out-of-distribution datasets, indicating its robustness.
Limitations and Future Work
- Computational Cost of Dynamic Kernels: While ContMix is designed for efficiency, dynamic kernel generation inherently adds some computational overhead compared to static kernels. The paper notes that ContMix achieves linear-time complexity by setting $S=7$.
- Further Exploration of DDS: The paper suggests that the DDS mechanism could be further explored and integrated with other network architectures.
- Optimizing ContMix Parameters: While the paper explores various ContMix configurations, further fine-tuning of parameters like the number of groups and kernel sizes could potentially yield even better results.
- Scaling to Larger Models: The paper presents variants up to "Base," and future work could explore even larger-scale models and their performance.
Summary updated