ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer

Zimian Wei, Tianqi Zhang, Hengyue PanarXiv 2026 · arXiv

TL;DR

ACSAN (All-Convolutional Self-Attention Network) is a "best of both worlds" architecture that replaces the heavy matrix multiplications of Vision Transformers (ViTs) with efficient convolutional operations. By integrating multi-scale dilated convolutions into a self-attention framework, it achieves superior accuracy on image classification tasks while significantly reducing computational overhead and the need for massive pre-training data.

What Problem Does This Paper Address?

Since their introduction, Vision Transformers (ViTs) have challenged the dominance of Convolutional Neural Networks (CNNs) in computer vision. However, ViTs come with a "data tax." Because they lack the inductive bias of CNNs—the inherent understanding that nearby pixels are related—they typically require massive datasets (like ImageNet-21k) and extensive pre-training to perform well.

Furthermore, ViTs are computationally expensive. The core of a Transformer relies on large-scale matrix multiplications in both the self-attention layers and the Multi-Layer Perceptrons (MLPs). This makes them difficult to deploy in resource-constrained environments or on smaller datasets where training from scratch is the only option. The authors of ACSAN aim to bridge this gap by injecting convolutional priors directly into the Transformer's DNA.

Methods

The authors introduce two primary innovations: the Dynamic Convolutional Self-Attention (DCSA) module and the ACSAN architecture.

The DCSA Module

In a standard ViT, the Query (Q), Key (K), and Value (V) vectors are generated using fully-connected linear layers. DCSA replaces these with multi-scale dilated convolutions.

As shown in Figure 2, the input is processed through three parallel branches with different dilation rates (3, 5, and 7). This allows the model to capture local features at multiple scales simultaneously.

Figure 2: Structure of the Dynamic Convolutional Self-Attention (DCSA) module. The DCSA module generates Q, K, V via multi-scale dilated convolutions, then applies parallel spatial and channel attention branches before fusion.

After generating Q, K, and V, the module splits into two parallel attention paths:

  1. Spatial Self-Attention: Captures global dependencies between different spatial locations (pixels).
  2. Channel Self-Attention: Captures dependencies between different feature maps (channels).

These two perspectives are merged to form the final output of the module.

The ACSAN Architecture

The overall network follows a hierarchical, four-stage design similar to modern CNNs or the Swin Transformer. It starts with a convolutional "stem" to process the raw image, followed by stages of DCSA blocks. Crucially, the authors also replace the standard MLPs found in Transformers with Equivalent Convolutional Fully Connected (EqConvFC) layers, making the entire network "all-convolutional."

Figure 1: Overall architecture of the proposed ACSAN model. The model consists of a convolution stem followed by multiple stages of DCSA Blocks with downsampling between stages.

Datasets

The authors evaluated ACSAN by training it from scratch (no pre-training) on three standard benchmarks:

  • CIFAR-10: 60,000 low-resolution images across 10 classes.
  • CIFAR-100: Similar to CIFAR-10 but with 100 classes, making it a more challenging "small data" task.
  • ImageNet-1k: The gold standard for large-scale image classification, containing over 1.2 million images.

Key Insights and Ideas

  • Convolutions as Feature Generators: By using convolutions to generate Q, K, and V, the model gains a "locality bias" that helps it learn much faster on smaller datasets compared to standard ViTs.
  • Efficiency through Sparsity: Convolutions are inherently more efficient than dense matrix multiplications. This allows ACSAN to achieve similar or better results than ViTs with a fraction of the Floating Point Operations (FLOPs).
  • The Power of Dual Attention: The ablation studies showed that combining spatial and channel attention provides a significant boost over using either one alone, suggesting that the model benefits from looking at "where" things are and "what" features are firing simultaneously.
  • No Pre-training Required: Perhaps the most significant insight is that ACSAN can be trained effectively from scratch on ImageNet, reaching 82.8% accuracy—a feat that standard ViTs struggle to achieve without massive external data.

Key Results

The quantitative results highlight a massive leap in efficiency:

  • CIFAR-10: ACSAN_base (9.75M parameters) achieved 93.52% accuracy using only 58.87M FLOPs. In contrast, a comparable ViT_base (9.82M parameters) reached only 90.50% accuracy while requiring a staggering 661.34M FLOPs.
  • CIFAR-100: ACSAN maintained a lead of roughly 8% in accuracy over standard ViT counterparts.
  • ImageNet-1k: ACSAN_ImageNet_large achieved 82.8% Top-1 accuracy. While this is competitive with state-of-the-art models like Swin-T (81.3%) and ConvNext-T (82.1%), ACSAN does so with significantly lower computational costs than the standard ViT_large (10.82G FLOPs vs 22.40G FLOPs).

Limitations and Future Work

While the paper demonstrates excellent results for image classification, it does not explore downstream tasks such as object detection or semantic segmentation. These tasks often require different architectural tweaks (like feature pyramid networks).

Additionally, the authors noted that the ImageNet experiments were conducted with a very small batch size (8) due to hardware constraints. It is possible that training with larger batches or more advanced augmentation strategies could push the performance of ACSAN even further. Future work could also investigate how this all-convolutional attention mechanism scales to even larger models or different modalities like video.

Summary updated

Abstract

In the past few years, Vision Transformer (ViT) has achieved excellent performance in the field of computer vision. However, there still some problems remain. For instance, ViTs heavily rely on large-scale training data since they simply contain a large number of trainable parameters, which slows down their learning efficiency and limits their applicability in resource-constrained scenarios. In this paper, we propose a novel deep learning model, namely All-Convolutional Self-Attention Network (ACSAN), which applies convolution operations to replace fully-connected matrices multiplications in the ViT framework. To implement ACSAN, we firstly design Dynamic Convolutional Self-Attention (DCSA) as the basic module, which employs multi-scale dilated convolutions to generate Query, Key, and Value. We defined spatial self-attention and channel self-attention to generate attention features respectively. The two attention features are finally merged as the output of the DCSA module. Based on DCSA, we then construct the basic ACSAN block. Comparing with ViT, we use DCSA to replace the regular self-attention operations, and use convolution layers to replace MLPs. To evaluate the performance of the proposed method, we train ACSAN on three widely-used image classification databases, i.e. CIFAR-10, CIFAR-100 and ImageNet-1k, from scratch. Experimental results show that ACSAN achieves excellent classification accuracy comparing with standard ViT with comparable or fewer parameters, as well as other state-of-the-art methods. Ablation studies verify that each component contributes positively to the final performance.
BibTeX
@misc{wei2026acsan,
  author = {Zimian Wei and Tianqi Zhang and Hengyue Pan},
  title = {{ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer}},
  year = {2026},
  eprint = {2610.35375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35375},
}
Figure