What Makes for Hierarchical Vision Transformer?

Yuxin Fang, Xinggang Wang, Rui Wu, Jianwei Niu, Wenyu LiuIEEE TPAMI 2021 · DOI · OpenReview

TL;DR

This paper investigates what truly drives the success of hierarchical Vision Transformers (ViTs) like Swin Transformer. By replacing the complex self-attention mechanism within local windows with simple linear layers (creating LINMAPPER), they demonstrate that the macro architecture (hierarchical structure with windowed attention and cross-window communication) is more critical than the specific aggregation method. This finding suggests a new path for efficient and transferable visual representation learning.

What Problem Does This Paper Address?

Hierarchical Vision Transformers (ViTs), particularly models like Swin Transformer, have achieved impressive results in computer vision, challenging the dominance of Convolutional Neural Networks (CNNs). These hierarchical ViTs typically employ a macro architecture that involves processing features in non-overlapped local windows using self-attention, followed by mechanisms to communicate information between these windows (e.g., shifted windows). While this architecture has proven effective, most follow-up research has focused on optimizing the cross-window communication strategies, largely taking the self-attention mechanism within windows as a given. This paper questions this assumption, asking: is self-attention truly essential for strong performance within these local windows, or could simpler mechanisms suffice? They aim to disentangle the contribution of the overall hierarchical structure from the specific choice of aggregation layers.

Methods

The core of this paper's methodology is the LINMAPPER architecture. LINMAPPER is a proof-of-concept model that replaces the Multi-Head Self-Attention (MHSA) layers within the local windows of existing hierarchical ViT architectures (specifically Swin Transformer, Shuffle Transformer, and MSG Transformer) with simple linear mapping layers. These linear layers perform a computationally less intensive aggregation of features within each window. The authors then systematically evaluate LINMAPPER across various tasks and configurations. They also explore variations of linear mapping, such as depth-wise linear mapping, and compare it against Multi-Layer Perceptrons (MLPs) for intra-window aggregation. The cross-window communication mechanisms (shifted windows, spatial token shuffle, messenger tokens) are kept consistent with the original models to isolate the effect of the aggregation layer.

Datasets

The primary dataset used for pre-training and evaluation is ImageNet-1k. For transfer learning experiments, the MS-COCO dataset is used for object detection and instance segmentation tasks.

Key Insights and Ideas

  • Macro Architecture is King: The most significant finding is that the hierarchical structure with interleaved window-based processing and cross-window communication is the primary driver of performance, not necessarily the complex self-attention mechanism within windows.
  • Simplicity Can Be Sufficient: Embarrassingly simple linear mapping layers can achieve competitive performance, challenging the notion that sophisticated attention is always required for local feature aggregation.
  • LINMAPPER's Transferability: The LINMAPPER architecture, being inherently linear, demonstrates excellent transferability to downstream dense prediction tasks like object detection and instance segmentation, even with less sophisticated pre-training. This is attributed to its ability to process arbitrary input resolutions.
  • Rethinking Attention's Role: The paper encourages a re-evaluation of the role of attention in ViTs, suggesting that its power might be more in enabling global interactions (when used across windows) rather than in local aggregation where simpler methods might suffice.

Key Results

  • ImageNet-1k Performance: LINMAPPER variants achieved competitive Top-1 accuracy on ImageNet-1k, often on par with or slightly below their MHSA-based counterparts, despite using significantly simpler aggregation layers. For example, Swin LINMAPPER-Tiny achieved 80.5% Top-1 accuracy, very close to Swin Transformer-Tiny's 81.3%.
  • Efficiency: LINMAPPER layers are more lightweight than MHSA layers in terms of parameters and computations, allowing for potentially deeper or wider architectures within similar computational budgets.
  • Transfer Learning: On MS-COCO, LINMAPPER demonstrated transfer learning performance comparable to Swin Transformer, even when pre-trained on ImageNet-1k with a less optimal setup.
  • Ablation Studies: Depth-wise linear layers did not offer significant improvements over shared linear layers, and MLPs did not provide further gains over simple linear mappings for intra-window aggregation.

Limitations and Future Work

  • MHSA Still Superior (in some aspects): The authors explicitly state that MHSA still outperforms linear mapping and MLPs in terms of raw accuracy, especially with fewer computational resources (referencing Table 7). The goal was not to prove linear layers are better, but to highlight the importance of the macro architecture.
  • Scaling Laws for MLPs: Finding principled scaling laws for MLP-based architectures like LINMAPPER is noted as a challenging area for future research, as their complexity differs from CNNs.
  • Sophisticated Model Scaling: The study of more sophisticated model scaling laws for LINMAPPER is left for future work.
  • Broader Exploration: While the paper explores different cross-window communication methods, a more exhaustive study of all possible combinations of aggregation layers and communication strategies could be beneficial.
  • New Architectures: The authors hope their work will inspire further research into general visual representation learning and encourage rethinking the design of Vision Transformers.

Summary updated

Abstract

Recent studies indicate that hierarchical Vision Transformer with a macro architecture of interleaved non-overlapped window-based self-attention \& shifted-window operation is able to achieve state-of-the-art performance in various visual recognition tasks, and challenges the ubiquitous convolutional neural networks (CNNs) using densely slid kernels. Most follow-up works attempt to replace the shifted-window operation with other kinds of cross-window communication paradigms, while treating self-attention as the de-facto standard for window-based information aggregation. In this manuscript, we question whether self-attention is the only choice for hierarchical Vision Transformer to attain strong performance, and the effects of different kinds of cross-window communication. To this end, we replace self-attention layers with embarrassingly simple linear mapping layers, and the resulting proof-of-concept architecture termed as LinMapper can achieve very strong performance in ImageNet-1k image recognition. Moreover, we find that LinMapper is able to better leverage the pre-trained representations from image recognition and demonstrates excellent transfer learning properties on downstream dense prediction tasks such as object detection and instance segmentation. We also experiment with other alternatives to self-attention for content aggregation inside each non-overlapped window under different cross-window communication approaches, which all give similar competitive results. Our study reveals that the \textbf{macro architecture} of Swin model families, other than specific aggregation layers or specific means of cross-window communication, may be more responsible for its strong performance and is the real challenger to the ubiquitous CNN's dense sliding window paradigm. Code and models will be publicly available to facilitate future research.
BibTeX
@article{fang2021what,
  author = {Yuxin Fang and Xinggang Wang and Rui Wu and Jianwei Niu and Wenyu Liu},
  title = {{What Makes for Hierarchical Vision Transformer?}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  doi = {10.1109/tpami.2023.3282019},
}
Figure