Abstract
ConvFormer introduces a novel Dynamic Multi-level Attention (DMA) mechanism that replaces the heavy self-attention of Vision Transformers with a more efficient, convolutional-based alternative. By combining multi-scale dilated convolutions with an input-adaptive gating mechanism, ConvFormer achieves state-of-the-art performance on ImageNet-1K and ADE20K, outperforming similar-sized models like Swin Transformer and ConvNeXt.
Introduction
The computer vision landscape is currently dominated by two primary architectures: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). While ViTs excel at capturing global context, they suffer from high computational costs due to self-attention scaling quadratically with image resolution. Conversely, CNNs are efficient and possess strong inductive biases like local correlation, but often lack the dynamic, data-specific adaptability inherent in Transformers.
Recent attempts to bridge this gap, such as ConvNeXt, have brought Transformer-like macro-designs to CNNs. However, these models often overlook multi-level feature extraction or lack "dynamic prosperity"—the ability to adapt weights based on specific input data. ConvFormer addresses these shortcomings by introducing a module that maintains convolutional efficiency while providing Transformer-like adaptivity.
Methodology
The core of ConvFormer is the Dynamic Multi-level Attention (DMA) module, integrated into a standard four-stage hierarchical backbone.
The DMA module utilizes a three-pronged approach to feature extraction:
- Multi-scale Dilated Convolution: Input is processed through three parallel depth-wise convolution branches ($3 \times 3$, $5 \times 5$, and $7 \times 7$). To expand the receptive field without increasing parameter counts, the larger kernels use dilation rates of 2 and 3.
- Gating Mechanism: This provides the "dynamic" element. A lightweight sub-network—consisting of Global Average Pooling (GAP), two Fully Connected (FC) layers, and a Sigmoid function—generates an attention vector to re-calibrate fused features based on the input.
- Hierarchical Backbone: ConvFormer follows a macro-design similar to Swin Transformer or ResNet, utilizing MLP blocks and LayerNorm across four stages.
By using dilated convolutions in a multi-scale setup, the model achieves a large effective receptive field similar to ViTs without the quadratic complexity of global self-attention.
Experimental Results
ConvFormer was evaluated on ImageNet-1K for classification and ADE20K for semantic segmentation.
ImageNet-1K Classification
ConvFormer consistently outperformed established competitors in both accuracy and parameter efficiency:
- ConvFormer-S (26.7M parameters) achieved 82.8% Top-1 accuracy, surpassing Swin-T (81.3%) and ConvNeXt-T (82.1%).
- ConvFormer-L (45.0M parameters) reached 83.6%, outperforming Swin-S and ConvNeXt-S while utilizing approximately 10% fewer parameters.
ADE20K Semantic Segmentation
When integrated into the UperNet framework, ConvFormer-S achieved 47.6 mIoU, notably higher than ConvNeXt-T (46.7) and Swin-T (46.1). Visual interpretability via Grad-CAM further confirmed that ConvFormer focuses more precisely on target objects compared to the more "scattered" attention maps of ResNet50 or Swin-T.
Discussion and Conclusion
ConvFormer demonstrates that actual dot-product attention is not strictly necessary for Transformer-level performance. By combining the macro-design of Transformers with optimized convolutional micro-designs, superior efficiency and adaptivity can be achieved.
Ablation studies highlight the necessity of the gating mechanism; without it, the model's ability to selectively emphasize informative features diminishes, leading to lower accuracy. While highly effective, future work could explore hardware-specific optimizations for parallel dilated branches and extend the architecture to video processing and object detection tasks.
Summary updated