On the Integration of Self-Attention and Convolution

Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, Gao Huang

TL;DR

Researchers have discovered that convolution and self-attention—long considered distinct architectural rivals—actually share the same "heavy lifting" operations. By sharing a common set of $1 \times 1$ convolutions and then splitting into lightweight aggregation paths, the proposed ACmix module combines the local inductive bias of CNNs with the dynamic flexibility of Transformers for nearly the same computational cost as a single model.

What Problem Does This Paper Address?

In computer vision, we’ve seen a tug-of-war between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). CNNs are great because they have a "local" bias—they know that nearby pixels are related—but they are rigid. Self-attention (the core of Transformers) is "dynamic" and can capture long-range dependencies, but it’s computationally expensive and lacks the inherent structural knowledge of images.

Previous attempts to combine them usually involved stacking them or running them in parallel, which often doubles the computational overhead. The authors of this paper asked a fundamental question: Are these two operations actually more similar than they look? If they share a mathematical foundation, can we integrate them without the "double-tax" on performance?

Methods

The authors decompose both convolution and self-attention into two distinct stages:

  1. Stage I (Feature Projection): Both paradigms start by projecting input features into a deeper space. In a $k \times k$ convolution, this is equivalent to $k^2$ individual $1 \times 1$ convolutions. In self-attention, this is the projection of Queries, Keys, and Values. Crucially, this stage accounts for the vast majority of the computational cost (quadratic with respect to channel size).
  2. Stage II (Aggregation): This is where they diverge. Convolution "shifts" the features and sums them up. Self-attention computes attention weights and aggregates the "Values." This stage is relatively lightweight (linear with respect to channel size).

The ACmix Module: The authors propose performing the heavy Stage I projections just once. The resulting intermediate features are then shared by two paths:

  • The Convolution Path: Uses a light fully connected layer to generate "shift" kernels, followed by an efficient "shift and summation" (implemented via group convolutions).
  • The Self-Attention Path: Follows the standard multi-head attention logic.

The outputs are combined using two learnable scalars ($\alpha$ and $\beta$), allowing the network to decide how much to trust each path at different depths.

Datasets

The model was rigorously tested on standard computer vision benchmarks:

  • ImageNet-1K: For image classification.
  • COCO: For object detection and instance segmentation (using RetinaNet, Mask R-CNN, and Cascade Mask R-CNN heads).
  • ADE20K: For semantic segmentation (using Semantic FPN and UperNet).

The authors integrated ACmix into several backbones, including ResNet, SAN, PVT, and Swin-Transformer, to show its versatility.

Key Insights and Ideas

  • The Hidden Marriage: The most profound insight is that $1 \times 1$ convolutions are the common denominator. By recognizing this, the authors turned a "parallel path" problem into a "shared trunk" problem.
  • Efficiency Bottleneck: The paper quantifies that in a standard $3 \times 3$ convolution, 99% of the computation happens in the projection stage. For self-attention, it's about 83%. This justifies why sharing Stage I is such a massive win for efficiency.
  • Dynamic Preference: By looking at the learned $\alpha$ and $\beta$ weights, the authors found a pattern: the model prefers convolution in the early layers (to catch local textures) and shifts its preference toward self-attention in the later layers (to understand global context).
  • Practical Speedups: Shifting tensors in memory is theoretically "free" but practically slow on GPUs. The authors solved this by implementing the shift operation as a group convolution with fixed kernels, significantly boosting inference speed.

Key Results

ACmix consistently outperformed pure CNN or Transformer baselines across the board:

  • ImageNet: ResNet-ACmix-50 achieved 77.8% top-1 accuracy, outperforming the standard ResNet-50 (76.9%) while using fewer FLOPs and parameters.
  • Segmentation: On ADE20K, ACmix improved the PVT-T backbone's mIoU by a staggering +5.6 points.
  • Detection: On COCO, ACmix models consistently beat their counterparts by +1.0 to +3.0 mAP.
  • Inference Speed: Despite being a hybrid, ACmix is highly efficient. For example, ACmix-L achieved 1.8x the FPS of a comparable PVT-L model while maintaining higher accuracy.

Limitations and Future Work

While ACmix is highly effective, the paper notes a few areas for further exploration:

  • Complexity of Aggregation: The current Stage II aggregation is kept simple to maintain efficiency. There may be room for more sophisticated, yet still lightweight, aggregation functions.
  • Hyperparameter Sensitivity: The model introduces $N$ (number of heads) and $k$ (kernel size) as parameters that need tuning, though the authors found $k=3$ and $N=4$ to be generally robust.
  • Broader Modalities: The study focuses entirely on 2D images. Applying this "shared projection" logic to 3D point clouds or video data remains an open research direction.

Summary updated

Abstract

Convolution and self-attention are two powerful techniques for representation learning, and they are usually considered as two peer approaches that are distinct from each other. In this paper, we show that there exists a strong underlying relation between them, in the sense that the bulk of computations of these two paradigms are in fact done with the same operation. Specifically, we first show that a traditional convolution with kernel size k x k can be decomposed into k^2 individual 1x1 convolutions, followed by shift and summation operations. Then, we interpret the projections of queries, keys, and values in self-attention module as multiple 1x1 convolutions, followed by the computation of attention weights and aggregation of the values. Therefore, the first stage of both two modules comprises the similar operation. More importantly, the first stage contributes a dominant computation complexity (square of the channel size) comparing to the second stage. This observation naturally leads to an elegant integration of these two seemingly distinct paradigms, i.e., a mixed model that enjoys the benefit of both self-Attention and Convolution (ACmix), while having minimum computational overhead compared to the pure convolution or self-attention counterpart. Extensive experiments show that our model achieves consistently improved results over competitive baselines on image recognition and downstream tasks. Code and pre-trained models will be released at https://github.com/LeapLabTHU/ACmix and https://gitee.com/mindspore/models.
BibTeX
@inproceedings{pan2021integration,
  author = {Xuran Pan and Chunjiang Ge and Rui Lu and Shiji Song and Guanfu Chen and Zeyi Huang and Gao Huang},
  title = {{On the Integration of Self-Attention and Convolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr52688.2022.00089},
}
Figure