OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels

Meng Lou, Yizhou Yu

TL;DR

This paper introduces OverLoCK, a novel ConvNet architecture that mimics human vision's top-down attention mechanism. By decomposing the network into "overview" and "look closely" stages and employing a new "context-mixing dynamic convolution" (ContMix), OverLoCK achieves state-of-the-art performance on image classification, object detection, and semantic segmentation with improved efficiency.

What Problem Does This Paper Address?

Modern Convolutional Neural Networks (CNNs) typically process images through a hierarchical structure that progressively downsamples features, expanding receptive fields. While effective, this approach neglects a key principle of human vision: the brain first forms a rough "overview" of a scene to identify salient cues, then performs a more detailed "look closely" analysis. Existing CNNs struggle to explicitly incorporate this top-down attention, leading to limitations in accurately localizing objects and capturing fine-grained details, especially at higher resolutions. This paper aims to bridge this gap by designing a ConvNet architecture that explicitly leverages top-down contextual guidance.

Methods

The core of OverLoCK lies in its Deep-stage Decomposition Strategy (DDS), which splits the network into three synergistic sub-networks:

  1. Base-Net: Encodes low- and mid-level features, similar to traditional CNN backbones.
  2. Overview-Net: A lightweight network that rapidly generates a semantically meaningful but low-quality "overview" feature map from the Base-Net's output. This serves as a "context prior."
  3. Focus-Net: A more powerful network that refines features and expands receptive fields, guided by the context prior from Overview-Net.

To effectively utilize this top-down guidance, OverLoCK introduces a novel Context-Mixing Dynamic Convolution (ContMix). ContMix dynamically generates convolution kernels for each token by considering its affinity with regional centers in the context prior. This allows the kernels to capture long-range dependencies while preserving the inherent local inductive biases of convolutions, a capability lacking in previous dynamic convolution methods. The architecture is further enhanced by a gated mechanism in the Focus-Net's Gated Dynamic Spatial Aggregator (GDSA) to filter contextual noise.

Datasets

The paper uses the following datasets:

  • ImageNet-1K: For pre-training and evaluating image classification performance.
  • COCO 2017: For evaluating object detection and instance segmentation.
  • ADE20K: For evaluating semantic segmentation.
  • ImageNet-V2, ImageNet-A, ImageNet-R, ImageNet-Sketch: For robustness evaluation.

Key Insights and Ideas

  • Biomimicry for CNNs: The paper demonstrates that mimicking human visual attention mechanisms (overview-first, look-closely-next) can lead to significant improvements in CNN performance and efficiency.
  • Decomposition for Context: Splitting the network into distinct "overview" and "focus" stages allows for explicit top-down contextual guidance, which is crucial for accurate perception.
  • Dynamic Kernels for Long-Range Dependencies: ContMix effectively addresses the limitation of fixed convolutional kernels by generating context-aware, dynamic kernels that can capture long-range spatial relationships without sacrificing local inductive biases.
  • ConvNets Can Compete with Transformers: OverLoCK shows that pure ConvNet architectures, when designed with advanced mechanisms like DDS and ContMix, can achieve performance competitive with, and often superior to, Transformer and Mamba-based models, especially in terms of efficiency.

Key Results

  • Image Classification: OverLoCK-T achieves 84.2% Top-1 accuracy on ImageNet-1K, outperforming ConvNeXt-B with significantly fewer FLOPs and parameters (approximately one-third). OverLoCK-B reaches 85.1% Top-1 accuracy.
  • Object Detection: OverLoCK-S surpasses MogaNet-B by 1% in AP$^b$ on the COCO dataset using Cascade Mask R-CNN.
  • Semantic Segmentation: OverLoCK-T improves UniRepLKNet-T by 1.7% in mIoU on ADE20K using UperNet.
  • Efficiency: OverLoCK demonstrates an excellent speed-accuracy trade-off, often achieving higher accuracy at faster inference speeds compared to other models. For example, OverLoCK-S surpasses MogaNet-B by over 100 imgs/s.
  • Robustness: OverLoCK shows strong performance on out-of-distribution datasets, indicating its robustness.

Limitations and Future Work

  • Computational Cost of Dynamic Kernels: While ContMix is designed for efficiency, dynamic kernel generation inherently adds some computational overhead compared to static kernels. The paper notes that ContMix achieves linear-time complexity by setting $S=7$.
  • Further Exploration of DDS: The paper suggests that the DDS mechanism could be further explored and integrated with other network architectures.
  • Optimizing ContMix Parameters: While the paper explores various ContMix configurations, further fine-tuning of parameters like the number of groups and kernel sizes could potentially yield even better results.
  • Scaling to Larger Models: The paper presents variants up to "Base," and future work could explore even larger-scale models and their performance.

Summary updated

Abstract

In the human vision system, top-down attention plays a crucial role in perception, wherein the brain initially performs an overall but rough scene analysis to extract salient cues (i.e., overview first), followed by a finer-grained examination to make more accurate judgments (i.e., look closely next). However, recent efforts in ConvNet designs primarily focused on increasing kernel size to obtain a larger receptive field without considering this crucial biomimetic mechanism to further improve performance. To this end, we propose a novel pure ConvNet vision backbone, termed OverLoCK, which is carefully devised from both the architecture and mixer perspectives. Specifically, we introduce a biomimetic Deep-stage Decomposition Strategy (DDS) that fuses semantically meaningful context representations into middle and deep layers by providing dynamic top-down context guidance at both feature and kernel weight levels. To fully unleash the power of top-down context guidance, we further propose a novel \textbf{Cont}ext-\textbf{Mix}ing Dynamic Convolution (ContMix) that effectively models long-range dependencies while preserving inherent local inductive biases even when the input resolution increases. These properties are absent in previous convolutions. With the support from both DDS and ContMix, our OverLoCK exhibits notable performance improvement over existing methods. For instance, OverLoCK-T achieves a Top-1 accuracy of 84.2\%, significantly surpassing ConvNeXt-B while only using around one-third of the FLOPs/parameters. On object detection with Cascade Mask R-CNN, our OverLoCK-S surpasses MogaNet-B by a significant 1\% in AP$^b$. On semantic segmentation with UperNet, our OverLoCK-T remarkably improves UniRepLKNet-T by 1.7\% in mIoU. Code is publicly available at https://github.com/LMMMEng/OverLoCK.
BibTeX
@inproceedings{lou2025overlock,
  author = {Meng Lou and Yizhou Yu},
  title = {{OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00021},
}
Figure