SimViT: Exploring a Simple Vision Transformer with Sliding Windows

Gang Li, Di Xu, Xing Cheng, Lingyu Si, Changwen Zheng

TL;DR

This paper introduces SimViT, a novel vision Transformer architecture that effectively integrates local spatial information and global context by employing sliding windows and a new Multi-head Central Self-Attention (MCSA) mechanism. SimViT achieves state-of-the-art performance on various vision tasks while being remarkably parameter-efficient, with its smallest variant being the most compact vision Transformer to date.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have shown great promise in computer vision, but they often struggle to preserve the inherent spatial and local correlations present in 2D image structures. This is because they typically flatten image patches into 1D tokens and rely on positional embeddings, which can disrupt local relationships and lack translation invariance. Convolutional Neural Networks (CNNs), on the other hand, excel at capturing local features due to their inherent sliding window operations and regional connections. This paper aims to bridge this gap by developing a vision Transformer that combines the strengths of both CNNs and ViTs, specifically by incorporating spatial structure and local information into the Transformer architecture.

Methods

The core innovation of SimViT lies in its architecture and a new attention mechanism:

  • Sliding Windows: SimViT utilizes overlapping sliding windows to aggregate spatial information and establish cross-window connections. This approach mimics the local receptive fields of CNNs, helping to preserve spatial structure.
  • Multi-head Central Self-Attention (MCSA): Instead of the standard Multi-head Self-Attention (MSA) that computes attention over all tokens in a window, MCSA focuses on the central patch. The feature of the central patch is used as the query, while the features of surrounding patches within the window serve as keys and values. This allows for capturing highly local relationships efficiently.
  • Hierarchical Structure: SimViT adopts a multi-scale hierarchical structure, similar to CNNs, with four stages. The first three stages employ MCSA blocks to build local relations and extract low-level features. The final stage uses conventional MSA blocks to capture global dependencies, enabling the model to integrate both local and global visual information.
  • Convolutional Feed Forward Network (ConvFFN): SimViT incorporates a ConvFFN, which adds a depth-wise convolution within the feed-forward network of the Transformer block, further enhancing local feature extraction.

Datasets

The paper evaluates SimViT on the following datasets:

  • ImageNet-1k: Used for image classification tasks.
  • COCO 2017: Used for object detection tasks.
  • ADE20K: Used for semantic segmentation tasks.

Key Insights and Ideas

  • Bridging the CNN-ViT Gap: SimViT successfully demonstrates that by incorporating CNN-like mechanisms (sliding windows, local attention) into a Transformer architecture, one can achieve both strong performance and parameter efficiency, suggesting a path towards unifying these two powerful model families.
  • MCSA for Local Relations: The MCSA mechanism is a key enabler for capturing fine-grained local dependencies within windows, which is crucial for understanding image structure without relying heavily on explicit positional encodings.
  • Translation Invariance: By avoiding explicit positional encodings and leveraging local attention with sliding windows, SimViT naturally exhibits translation invariance, a desirable property for many vision tasks.
  • Parameter Efficiency: The design choices, particularly MCSA and the hierarchical structure, lead to remarkably compact models that achieve competitive or superior performance compared to larger models.

Key Results

  • Image Classification: SimViT-Micro, with only 3.3M parameters, achieves 71.1% top-1 accuracy on ImageNet-1k, making it the smallest vision Transformer model to date. SimViT-Tiny and SimViT-Small also show excellent performance, outperforming comparable models in terms of accuracy and parameter count.
  • Object Detection: When used as a backbone for object detection on COCO 2017, SimViT-Small provides consistent gains (e.g., +1.3 ~ 2.4 box AP) over Swin-T.
  • Semantic Segmentation: On ADE20K, SimViT models achieve significant improvements in mIOU (e.g., +5.7% ~ 10.5%) compared to ResNet and PVT.
  • Ablation Studies: Experiments confirm that SimViT's performance is not solely due to increased depth but also to its novel architecture. They also show that explicit position encoding is not necessary for SimViT's success, further highlighting its translation invariance.

Limitations and Future Work

  • Model Depth vs. Architecture: While ablation studies show SimViT's depth contributes to performance, the paper acknowledges that ruling out the possibility of model depth solely causing superiority is complex. Future work could further disentangle these effects.
  • Window Size Impact: The paper notes that larger window sizes in MCSA did not necessarily improve results, suggesting an optimal window size might exist and warrants further investigation.
  • Broader Applicability: The authors express optimism that SimViT's effectiveness as a general-purpose backbone can be applied to a wider range of vision tasks beyond those tested.

Summary updated

Abstract

Although vision Transformers have achieved excellent performance as backbone models in many vision tasks, most of them intend to capture global relations of all tokens in an image or a window, which disrupts the inherent spatial and local correlations between patches in 2D structure. In this paper, we introduce a simple vision Transformer named SimViT, to incorporate spatial structure and local information into the vision Transformers. Specifically, we introduce Multi-head Central Self-Attention(MCSA) instead of conventional Multi-head Self-Attention to capture highly local relations. The introduction of sliding windows facilitates the capture of spatial structure. Meanwhile, SimViT extracts multiscale hierarchical features from different layers for dense prediction tasks. Extensive experiments show the SimViT is effective and efficient as a general-purpose backbone model for various image processing tasks. Especially, our SimViT-Micro only needs 3.3M parameters to achieve 71.1% top-1 accuracy on ImageNet-1k dataset, which is the smallest size vision Transformer model by now.
BibTeX
@inproceedings{li2021simvit,
  author = {Gang Li and Di Xu and Xing Cheng and Lingyu Si and Changwen Zheng},
  title = {{SimViT: Exploring a Simple Vision Transformer with Sliding Windows}},
  booktitle = {ICME},
  year = {2021},
  doi = {10.1109/icme52920.2022.9859907},
}
Figure