Rethinking the Self-Attention in Vision Transformers

Kyungmin Kim, Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Zhicheng Yan, Peter Vajda, Seon Joo KimWorkshops @ CVPR 2021 · OpenReview

Abstract

Self-attention is a corner stone for transformer models. However, our analysis shows that self-attention in vision transformer inference is extremely sparse. When applying a sparsity constraint, our experiments on image (ImageNet-1K) and video (Kinetics-400) understanding show we can achieve 95% sparsity on the self-attention maps while maintaining the performance drop to be less than 2 points. This motivates us to rethink the role of self-attention in vision transformer models.

Introduction

Vision Transformers (ViTs) have revolutionized computer vision but suffer from high computational costs due to their "compare everything to everything" self-attention mechanism. This approach imposes a "quadratic tax" on computation, creating a significant bottleneck as token counts increase for high-resolution images or long-form video. This paper investigates whether this exhaustive global attention is necessary, revealing that self-attention is inherently sparse and often redundant.

Analysis of Attention Sparsity

The researchers analyzed pre-trained models (DeiT for images, TimeSFormer for video) across millions of samples to characterize attention behaviors. They identified three primary types of attention heads:

  1. Relative Position-based: Heads that focus on immediate spatial neighbors, acting similarly to a convolution.
  2. Absolute Position-based: Heads that consistently fixate on specific locations, most notably the [CLS] token.
  3. Content-based: Heads that dynamically adapt based on the input image, typically focusing on salient objects.
Evolution of attention patterns across layers and heads. Early layers show more structured, local patterns, while deeper layers exhibit more global or content-specific behaviors.

A striking finding is that most elements in an attention map are near zero. In a typical model with 197 tokens, a head might attend to only about 9 tokens on average. Furthermore, attention behavior evolves through the network: early layers are generally more "local" or anchor-based, while deeper layers become more "global" and content-aware.

Extracted figure from page 2 of Rethinking the Self-Attention in Vision Transformers

Data-Driven Masking and Results

To leverage this sparsity, the authors introduced a masking mechanism to the attention computation. Among various strategies, Information-based masking proved most effective. This data-driven approach retains tokens until a specific threshold of the total "attention energy" (cumulative similarity score) is reached, allowing the model to dynamically choose how many tokens to attend to based on input complexity.

Experiments on ImageNet-1K and Kinetics-400 demonstrated that:

  • 95% Sparsity is Achievable: The authors maintained model performance within 2 percentage points of the original accuracy while pruning 95% of attention connections.
  • Robustness Across Modalities: The findings held true for both image classification and video action classification, with video models showing even greater potential for scaling via sparsity.

Discussion and Future Work

Despite achieving 95% sparsity in attention maps, current hardware speedups are limited by the Feed-Forward Network (FFN) Bottleneck. In architectures like DeiT-Base, FFN layers account for roughly 64% of total GFLOPs, while attention map computation is only about 4%.

To fully realize the benefits of this sparsity, future work must address FFN optimization or apply these masking techniques to models with much larger token counts—such as ultra-high-resolution images or long-form video—where attention computation begins to dominate the total computational budget.

Summary updated

Abstract

Self-attention is a corner stone for transformer models. However, our analysis shows that self-attention in vision transformer inference is extremely sparse. When applying a sparsity constraint, our experiments on image (ImageNet-1K) and video (Kinetics-400) understanding show we can achieve 95% sparsity on the self-attention maps while maintaining the performance drop to be less than 2 points. This motivates us to rethink the role of self-attention in vision transformer models.
BibTeX
@inproceedings{kim2021rethinking,
  author = {Kyungmin Kim and Bichen Wu and Xiaoliang Dai and Peizhao Zhang and Zhicheng Yan and Peter Vajda and Seon Joo Kim},
  title = {{Rethinking the Self-Attention in Vision Transformers}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=s7AuycGMtzA},
}
Figure