TL;DR
This paper investigates what truly drives the success of hierarchical Vision Transformers (ViTs) like Swin Transformer. By replacing the complex self-attention mechanism within local windows with simple linear layers (creating LINMAPPER), they demonstrate that the macro architecture (hierarchical structure with windowed attention and cross-window communication) is more critical than the specific aggregation method. This finding suggests a new path for efficient and transferable visual representation learning.
What Problem Does This Paper Address?
Hierarchical Vision Transformers (ViTs), particularly models like Swin Transformer, have achieved impressive results in computer vision, challenging the dominance of Convolutional Neural Networks (CNNs). These hierarchical ViTs typically employ a macro architecture that involves processing features in non-overlapped local windows using self-attention, followed by mechanisms to communicate information between these windows (e.g., shifted windows). While this architecture has proven effective, most follow-up research has focused on optimizing the cross-window communication strategies, largely taking the self-attention mechanism within windows as a given. This paper questions this assumption, asking: is self-attention truly essential for strong performance within these local windows, or could simpler mechanisms suffice? They aim to disentangle the contribution of the overall hierarchical structure from the specific choice of aggregation layers.
Methods
The core of this paper's methodology is the LINMAPPER architecture. LINMAPPER is a proof-of-concept model that replaces the Multi-Head Self-Attention (MHSA) layers within the local windows of existing hierarchical ViT architectures (specifically Swin Transformer, Shuffle Transformer, and MSG Transformer) with simple linear mapping layers. These linear layers perform a computationally less intensive aggregation of features within each window. The authors then systematically evaluate LINMAPPER across various tasks and configurations. They also explore variations of linear mapping, such as depth-wise linear mapping, and compare it against Multi-Layer Perceptrons (MLPs) for intra-window aggregation. The cross-window communication mechanisms (shifted windows, spatial token shuffle, messenger tokens) are kept consistent with the original models to isolate the effect of the aggregation layer.
Datasets
The primary dataset used for pre-training and evaluation is ImageNet-1k. For transfer learning experiments, the MS-COCO dataset is used for object detection and instance segmentation tasks.
Key Insights and Ideas
- Macro Architecture is King: The most significant finding is that the hierarchical structure with interleaved window-based processing and cross-window communication is the primary driver of performance, not necessarily the complex self-attention mechanism within windows.
- Simplicity Can Be Sufficient: Embarrassingly simple linear mapping layers can achieve competitive performance, challenging the notion that sophisticated attention is always required for local feature aggregation.
- LINMAPPER's Transferability: The LINMAPPER architecture, being inherently linear, demonstrates excellent transferability to downstream dense prediction tasks like object detection and instance segmentation, even with less sophisticated pre-training. This is attributed to its ability to process arbitrary input resolutions.
- Rethinking Attention's Role: The paper encourages a re-evaluation of the role of attention in ViTs, suggesting that its power might be more in enabling global interactions (when used across windows) rather than in local aggregation where simpler methods might suffice.
Key Results
- ImageNet-1k Performance: LINMAPPER variants achieved competitive Top-1 accuracy on ImageNet-1k, often on par with or slightly below their MHSA-based counterparts, despite using significantly simpler aggregation layers. For example, Swin LINMAPPER-Tiny achieved 80.5% Top-1 accuracy, very close to Swin Transformer-Tiny's 81.3%.
- Efficiency: LINMAPPER layers are more lightweight than MHSA layers in terms of parameters and computations, allowing for potentially deeper or wider architectures within similar computational budgets.
- Transfer Learning: On MS-COCO, LINMAPPER demonstrated transfer learning performance comparable to Swin Transformer, even when pre-trained on ImageNet-1k with a less optimal setup.
- Ablation Studies: Depth-wise linear layers did not offer significant improvements over shared linear layers, and MLPs did not provide further gains over simple linear mappings for intra-window aggregation.
Limitations and Future Work
- MHSA Still Superior (in some aspects): The authors explicitly state that MHSA still outperforms linear mapping and MLPs in terms of raw accuracy, especially with fewer computational resources (referencing Table 7). The goal was not to prove linear layers are better, but to highlight the importance of the macro architecture.
- Scaling Laws for MLPs: Finding principled scaling laws for MLP-based architectures like LINMAPPER is noted as a challenging area for future research, as their complexity differs from CNNs.
- Sophisticated Model Scaling: The study of more sophisticated model scaling laws for LINMAPPER is left for future work.
- Broader Exploration: While the paper explores different cross-window communication methods, a more exhaustive study of all possible combinations of aggregation layers and communication strategies could be beneficial.
- New Architectures: The authors hope their work will inspire further research into general visual representation learning and encourage rethinking the design of Vision Transformers.
Summary updated