Abstract
Vision Transformers (ViTs) have achieved impressive results in computer vision by leveraging self-attention to model long-range dependencies. However, their emphasis on global context often comes at the expense of local feature extraction in small datasets, particularly due to the lack of key inductive biases such as locality and translation equivariance. To mitigate this, we propose CoSwin, a novel feature-fusion architecture that augments the hierarchical shifted window attention with localized convolutional feature learning. Specifically, CoSwin integrates a learnable local feature enhancement module into each attention block, enabling the model to simultaneously capture fine-grained spatial details and global semantic structure. We evaluate CoSwin on multiple image classification benchmarks including CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet. Our experimental results show consistent performance gains over state-of-the-art convolutional and transformer-based models. Notably, CoSwin achieves improvements of 2.17% on CIFAR-10, 4.92% on CIFAR-100, 0.10% on MNIST, 0.26% on SVHN, and 4.47% on Tiny ImageNet over the baseline Swin Transformer. These improvements underscore the effectiveness of local-global feature fusion in enhancing the generalization and robustness of transformers for small-scale vision.
Introduction
While Vision Transformers (ViTs) excel at capturing global context through self-attention, they are notoriously "data-hungry." Unlike Convolutional Neural Networks (CNNs), standard ViTs lack inherent inductive biases like locality and translation equivariance. On massive datasets, ViTs can learn these properties from scratch, but on small-scale datasets such as CIFAR or MNIST, they often struggle to generalize. CoSwin addresses this by reintroducing local inductive biases through a hybrid architecture that fuses the hierarchical structure of the Swin Transformer with localized convolutional feature learning.
Architecture and Methodology
CoSwin follows a four-stage hierarchical design, integrating a learnable convolutional module directly into the Swin Transformer’s shifted window attention mechanism.
The core components of the CoSwin block include:
- Patch Conversion: Tokens are reshaped back into a 2D spatial format to allow standard convolutional operations to process the "imaginary image."
- Local Feature Enhancement: A two-layer $3 \times 3$ convolution module captures fine-grained spatial details.
- Attention Fusion: A learnable scalar weight ($\Gamma$) adaptively balances the contribution of local convolutional features and global attention context during training.
- Hierarchical Shifted Windows: The model maintains the Swin Transformer’s efficiency by computing attention within shifted local windows that move between layers.
By combining these elements, CoSwin leverages attention to reduce bias and convolution to reduce variance, resulting in a more robust training process for small datasets.
Experimental Results
The performance of CoSwin was evaluated across five benchmark datasets: CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet.
As shown in the results table, CoSwin consistently outperforms both pure CNNs (e.g., ResNet) and pure Transformers (e.g., SwinT). The most significant gains were observed on CIFAR-100 and Tiny ImageNet, where CoSwin improved upon the baseline Swin Transformer by 4.92% and 4.47%, respectively. Qualitative analysis using Grad-CAM further revealed that CoSwin exhibits a sharper visual focus on target objects compared to the more "blurry" or distracted attention maps of standard ViTs on small images.
Conclusion and Future Work
CoSwin demonstrates that integrating local convolutional biases into hierarchical transformers significantly enhances performance on small-scale vision tasks. The learnable fusion mechanism allows the model to dynamically prioritize local versus global information, leading to better generalization and faster convergence. Future research will explore scaling CoSwin to larger datasets and extending the architecture to dense prediction tasks like object detection and semantic segmentation, as well as domain-specific applications in medical imaging.
Code and pretrained weights are available at https://github.com/puskal-khadka/coswin.
Summary updated