TL;DR
Researchers have discovered that convolution and self-attention—long considered distinct architectural rivals—actually share the same "heavy lifting" operations. By sharing a common set of $1 \times 1$ convolutions and then splitting into lightweight aggregation paths, the proposed ACmix module combines the local inductive bias of CNNs with the dynamic flexibility of Transformers for nearly the same computational cost as a single model.
What Problem Does This Paper Address?
In computer vision, we’ve seen a tug-of-war between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). CNNs are great because they have a "local" bias—they know that nearby pixels are related—but they are rigid. Self-attention (the core of Transformers) is "dynamic" and can capture long-range dependencies, but it’s computationally expensive and lacks the inherent structural knowledge of images.
Previous attempts to combine them usually involved stacking them or running them in parallel, which often doubles the computational overhead. The authors of this paper asked a fundamental question: Are these two operations actually more similar than they look? If they share a mathematical foundation, can we integrate them without the "double-tax" on performance?
Methods
The authors decompose both convolution and self-attention into two distinct stages:
- Stage I (Feature Projection): Both paradigms start by projecting input features into a deeper space. In a $k \times k$ convolution, this is equivalent to $k^2$ individual $1 \times 1$ convolutions. In self-attention, this is the projection of Queries, Keys, and Values. Crucially, this stage accounts for the vast majority of the computational cost (quadratic with respect to channel size).
- Stage II (Aggregation): This is where they diverge. Convolution "shifts" the features and sums them up. Self-attention computes attention weights and aggregates the "Values." This stage is relatively lightweight (linear with respect to channel size).
The ACmix Module: The authors propose performing the heavy Stage I projections just once. The resulting intermediate features are then shared by two paths:
- The Convolution Path: Uses a light fully connected layer to generate "shift" kernels, followed by an efficient "shift and summation" (implemented via group convolutions).
- The Self-Attention Path: Follows the standard multi-head attention logic.
The outputs are combined using two learnable scalars ($\alpha$ and $\beta$), allowing the network to decide how much to trust each path at different depths.
Datasets
The model was rigorously tested on standard computer vision benchmarks:
- ImageNet-1K: For image classification.
- COCO: For object detection and instance segmentation (using RetinaNet, Mask R-CNN, and Cascade Mask R-CNN heads).
- ADE20K: For semantic segmentation (using Semantic FPN and UperNet).
The authors integrated ACmix into several backbones, including ResNet, SAN, PVT, and Swin-Transformer, to show its versatility.
Key Insights and Ideas
- The Hidden Marriage: The most profound insight is that $1 \times 1$ convolutions are the common denominator. By recognizing this, the authors turned a "parallel path" problem into a "shared trunk" problem.
- Efficiency Bottleneck: The paper quantifies that in a standard $3 \times 3$ convolution, 99% of the computation happens in the projection stage. For self-attention, it's about 83%. This justifies why sharing Stage I is such a massive win for efficiency.
- Dynamic Preference: By looking at the learned $\alpha$ and $\beta$ weights, the authors found a pattern: the model prefers convolution in the early layers (to catch local textures) and shifts its preference toward self-attention in the later layers (to understand global context).
- Practical Speedups: Shifting tensors in memory is theoretically "free" but practically slow on GPUs. The authors solved this by implementing the shift operation as a group convolution with fixed kernels, significantly boosting inference speed.
Key Results
ACmix consistently outperformed pure CNN or Transformer baselines across the board:
- ImageNet: ResNet-ACmix-50 achieved 77.8% top-1 accuracy, outperforming the standard ResNet-50 (76.9%) while using fewer FLOPs and parameters.
- Segmentation: On ADE20K, ACmix improved the PVT-T backbone's mIoU by a staggering +5.6 points.
- Detection: On COCO, ACmix models consistently beat their counterparts by +1.0 to +3.0 mAP.
- Inference Speed: Despite being a hybrid, ACmix is highly efficient. For example, ACmix-L achieved 1.8x the FPS of a comparable PVT-L model while maintaining higher accuracy.
Limitations and Future Work
While ACmix is highly effective, the paper notes a few areas for further exploration:
- Complexity of Aggregation: The current Stage II aggregation is kept simple to maintain efficiency. There may be room for more sophisticated, yet still lightweight, aggregation functions.
- Hyperparameter Sensitivity: The model introduces $N$ (number of heads) and $k$ (kernel size) as parameters that need tuning, though the authors found $k=3$ and $N=4$ to be generally robust.
- Broader Modalities: The study focuses entirely on 2D images. Applying this "shared projection" logic to 3D point clouds or video data remains an open research direction.
Summary updated