ConvFormer: Closing the Gap Between CNN and Vision Transformers

Zimian Wei, Hengyue Pan, Xin Niu, Dongsheng Li

Abstract

ConvFormer introduces a novel Dynamic Multi-level Attention (DMA) mechanism that replaces the heavy self-attention of Vision Transformers with a more efficient, convolutional-based alternative. By combining multi-scale dilated convolutions with an input-adaptive gating mechanism, ConvFormer achieves state-of-the-art performance on ImageNet-1K and ADE20K, outperforming similar-sized models like Swin Transformer and ConvNeXt.

Introduction

The computer vision landscape is currently dominated by two primary architectures: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). While ViTs excel at capturing global context, they suffer from high computational costs due to self-attention scaling quadratically with image resolution. Conversely, CNNs are efficient and possess strong inductive biases like local correlation, but often lack the dynamic, data-specific adaptability inherent in Transformers.

Recent attempts to bridge this gap, such as ConvNeXt, have brought Transformer-like macro-designs to CNNs. However, these models often overlook multi-level feature extraction or lack "dynamic prosperity"—the ability to adapt weights based on specific input data. ConvFormer addresses these shortcomings by introducing a module that maintains convolutional efficiency while providing Transformer-like adaptivity.

Methodology

The core of ConvFormer is the Dynamic Multi-level Attention (DMA) module, integrated into a standard four-stage hierarchical backbone.

Overall framework of ConvFormer and the structure of its basic block.

The DMA module utilizes a three-pronged approach to feature extraction:

  1. Multi-scale Dilated Convolution: Input is processed through three parallel depth-wise convolution branches ($3 \times 3$, $5 \times 5$, and $7 \times 7$). To expand the receptive field without increasing parameter counts, the larger kernels use dilation rates of 2 and 3.
  2. Gating Mechanism: This provides the "dynamic" element. A lightweight sub-network—consisting of Global Average Pooling (GAP), two Fully Connected (FC) layers, and a Sigmoid function—generates an attention vector to re-calibrate fused features based on the input.
  3. Hierarchical Backbone: ConvFormer follows a macro-design similar to Swin Transformer or ResNet, utilizing MLP blocks and LayerNorm across four stages.

By using dilated convolutions in a multi-scale setup, the model achieves a large effective receptive field similar to ViTs without the quadratic complexity of global self-attention.

Experimental Results

ConvFormer was evaluated on ImageNet-1K for classification and ADE20K for semantic segmentation.

Comparison of Top-1 accuracy vs. parameters on ImageNet-1K. ConvFormer (Ours) shows superior efficiency compared to Swin, ConvNeXt, and others.

ImageNet-1K Classification

ConvFormer consistently outperformed established competitors in both accuracy and parameter efficiency:

  • ConvFormer-S (26.7M parameters) achieved 82.8% Top-1 accuracy, surpassing Swin-T (81.3%) and ConvNeXt-T (82.1%).
  • ConvFormer-L (45.0M parameters) reached 83.6%, outperforming Swin-S and ConvNeXt-S while utilizing approximately 10% fewer parameters.

ADE20K Semantic Segmentation

When integrated into the UperNet framework, ConvFormer-S achieved 47.6 mIoU, notably higher than ConvNeXt-T (46.7) and Swin-T (46.1). Visual interpretability via Grad-CAM further confirmed that ConvFormer focuses more precisely on target objects compared to the more "scattered" attention maps of ResNet50 or Swin-T.

Discussion and Conclusion

ConvFormer demonstrates that actual dot-product attention is not strictly necessary for Transformer-level performance. By combining the macro-design of Transformers with optimized convolutional micro-designs, superior efficiency and adaptivity can be achieved.

Ablation studies highlight the necessity of the gating mechanism; without it, the model's ability to selectively emphasize informative features diminishes, leading to lower accuracy. While highly effective, future work could explore hardware-specific optimizations for parallel dilated branches and extend the architecture to video processing and object detection tasks.

Summary updated

Abstract

Vision transformers have shown excellent performance in computer vision tasks. As the computation cost of their self-attention mechanism is expensive, recent works tried to replace the self-attention mechanism in vision transformers with convolutional operations, which is more efficient with built-in inductive bias. However, these efforts either ignore multi-level features or lack dynamic prosperity, leading to sub-optimal performance. In this paper, we propose a Dynamic Multi-level Attention mechanism (DMA), which captures different patterns of input images by multiple kernel sizes and enables input-adaptive weights with a gating mechanism. Based on DMA, we present an efficient backbone network named DMFormer. DMFormer adopts the overall architecture of vision transformers, while replacing the self-attention mechanism with our proposed DMA. Extensive experimental results on ImageNet-1K and ADE20K datasets demonstrated that DMFormer achieves state-of-the-art performance, which outperforms similar-sized vision transformers(ViTs) and convolutional neural networks (CNNs).
BibTeX
@inproceedings{wei2023convformer,
  author = {Zimian Wei and Hengyue Pan and Xin Niu and Dongsheng Li},
  title = {{ConvFormer: Closing the Gap Between CNN and Vision Transformers}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10097256},
}
Figure