How Do Vision Transformers Work?

Namuk Park, Songkuk KimICLR 2022

While Vision Transformers (ViTs) have taken the computer vision world by storm, our understanding of why they work has remained somewhat superficial. We often point to "long-range dependencies" or "weak inductive biases" as the secret sauce. However, a paper from ICLR 2022 by Namuk Park and Songkuk Kim, titled "How Do Vision Transformers Work?", challenges these assumptions and provides a much more grounded, signal-processing-based explanation.

The Core Problem

The meteoric rise of ViTs is undeniable, yet they are notoriously difficult to train on small datasets (like CIFAR-100) compared to traditional Convolutional Neural Networks (CNNs). The common wisdom is that ViTs "overfit" because they lack the inductive bias of CNNs.

Park and Kim set out to verify if this is true. They ask: Is it really the long-range dependency that makes Multi-Head Self-Attention (MSA) powerful? Why do they struggle with small data? And most importantly, how do MSAs and Convolutions actually interact at a fundamental level?

Diagnostic Methods

To peer under the hood of these models, the authors employ several sophisticated diagnostic tools:

  • Hessian Max Eigenvalue Analysis: They calculate the "curviness" of the loss landscape. A smaller maximum eigenvalue indicates a flatter, more stable landscape that is easier to optimize and generalizes better.
  • Fourier Analysis: They examine the frequency components of feature maps to see if layers act as high-pass or low-pass filters.
  • Lesion Studies: They systematically remove layers from trained models to identify which parts of the architecture are most critical for prediction.
  • Centered Kernel Alignment (CKA): A method to measure the representational similarity between different layers and models.

Key Insights: Opposites Attract

The study reveals several surprising truths about how ViTs operate, fundamentally shifting how we view the relationship between attention and convolution.

MSAs are Low-Pass Filters; Convs are High-Pass This is the paper's "aha" moment. MSAs tend to reduce high-frequency signals (acting as spatial smoothing), while Convolutions (and MLPs) amplify them. This makes them fundamentally complementary: Convs capture the "texture" and fine details, while MSAs capture the "shape" and global structure.

It’s Not About Long-Range Dependency Contrary to popular belief, the authors show that data specificity—the ability of the attention weights to change based on the input—is more important than the ability to look across the whole image. In fact, local MSAs (with a restricted 3x3 or 5x5 window) often outperform global MSAs because they reduce unnecessary degrees of freedom that disrupt training.

ViTs Don't Overfit; They Fail to Optimize On small datasets, ViTs don't actually have lower training loss than CNNs (which would indicate overfitting). Instead, their loss landscapes are highly non-convex. Large datasets or strong data augmentation help "convexify" the loss, making it possible for ViTs to find good minima.

MSAs Flatten the Landscape The primary benefit of MSA is that it significantly reduces the magnitude of Hessian eigenvalues, leading to a flatter loss landscape. This is why adding even a little bit of attention to a CNN can boost its performance and robustness.

Loss landscapes and Hessian max eigenvalue spectra comparing ResNet and ViT.

AlterNet: The Best of Both Worlds

Based on these insights, the authors designed AlterNet, a hybrid model that strategically replaces the last Convolutional block of each stage in a ResNet with an MSA block.

  • Consistent Dominance: AlterNet consistently outperformed pure ResNets and ViTs across small (CIFAR) and large (ImageNet) datasets. For example, on CIFAR-100, AlterNet achieved significantly higher accuracy and robustness than ResNet-50.
  • Strategic Placement: Lesion studies revealed that MSAs are most effective when placed at the end of a stage. Removing an MSA at the end of a stage caused a much larger drop in accuracy than removing one at the beginning.
  • Robustness to Noise: Because MSAs act as low-pass filters, they are naturally more robust to high-frequency noise (like Gaussian noise) compared to CNNs, which are vulnerable to such perturbations.

Limitations and Future Directions

While AlterNet provides a powerful new baseline, the authors note a few areas for future exploration:

  • Uncertainty Calibration: The strong data augmentation required to make MSAs work well can harm "uncertainty calibration," meaning the models might become overconfident or underconfident in their predictions.
  • Dense Prediction Tasks: While the paper focuses on image classification, the ensembling effect of MSAs could be even more beneficial for dense tasks like object detection or segmentation.
  • Architecture Search: AlterNet was designed using manual "build-up rules." There is likely room for further optimization using automated neural architecture search (NAS) to find the perfect balance of Convs and MSAs.

Ultimately, this research demystifies the Vision Transformer, proving that its success isn't magic—it's just good signal processing.

Summary updated

BibTeX
@inproceedings{park2022how,
  author = {Namuk Park and Songkuk Kim},
  title = {{How Do Vision Transformers Work?}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
}
Figure