Make a Long Image Short: Adaptive Token Length for Vision Transformers

Qiqi Zhou, Yichen Zhu

TL;DR

This paper introduces a novel method to accelerate Vision Transformers (ViTs) by adaptively adjusting the number of image tokens processed during inference. By training a "Resizable-ViT" (ReViT) that can handle varying token lengths and a lightweight "Token-Length Assigner" (TLA), the authors achieve significant speedups with minimal accuracy loss, making ViTs more practical for real-world deployment.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have shown remarkable success in computer vision tasks, often outperforming traditional Convolutional Neural Networks (CNNs). However, a major bottleneck for ViTs is their high computational cost. This cost scales quadratically with the number of input tokens (image patches). While increasing the number of tokens generally improves accuracy, it drastically increases computational demands, making ViTs challenging to deploy on resource-constrained devices. The paper addresses this by proposing a way to use fewer tokens for "easy" images that can be classified with high confidence using minimal information, while still using sufficient tokens for more complex images.

Methods

The core of the proposed solution lies in a two-stage approach:

  1. Resizable-ViT (ReViT): This is a modified ViT architecture designed to process input images with a variable number of tokens. This is achieved through:
  2. Token-Length-Aware Layer Normalization (TAL-LN): Instead of sharing normalization statistics across different token lengths (which was found to be detrimental), ReViT uses independent LayerNorm layers for each predefined token length. This ensures proper normalization for each token configuration.
  3. Self-Distillation: To help smaller token-length models learn effectively, a self-distillation mechanism is employed. The model trained with the largest token length acts as a teacher, guiding the training of models with smaller token lengths. This is particularly important because reducing tokens can significantly hamper accuracy.
  4. Token-Length Assigner (TLA): This is a lightweight module trained after ReViT. Its purpose is to predict the optimal, minimum token length required for accurate classification of a given image at inference time. The TLA is trained using "token-length labels" derived from the ReViT, which indicate the smallest token length that yields a correct prediction for each image.

Training Strategy: * Mixing Token Length Training: To efficiently train ReViT, a strategy is used where a batch of images is processed with different token lengths, and gradients are accumulated before updating the model parameters. * Efficient Training Implementation: To further speed up training, a parallel computing strategy is employed. The model is replicated for each token length, and gradients are synchronized and accumulated on a central node. This reduces the training time complexity from O(k) to O(1), where k is the number of token lengths.

Datasets

The paper evaluates its approach on:

  • ImageNet: Used for image classification tasks.
  • Something-Something V2: Used for action recognition tasks.

Key Insights and Ideas

  • "Easy" vs. "Hard" Images: The observation that not all images require the same amount of information (tokens) for accurate classification is a key motivator. Some images are easily recognizable with very few patches, while others require more detailed analysis.
  • Dynamic Tokenization: The idea of making the tokenization process dynamic and input-dependent, rather than fixed, is central to reducing computational cost without sacrificing accuracy.
  • Decoupled TLA: Training a separate, lightweight TLA to guide the main ReViT model is an effective way to manage the complexity of assigning token lengths. This decouples the task of learning image features from the task of determining optimal input representation.
  • TAL-LN is Crucial: The finding that shared Layer Normalization across different token lengths is detrimental highlights the importance of specialized normalization for dynamic architectures.

Key Results

  • Image Classification:
    • Achieved a 50% acceleration on DeiT-S with only a 0.1% accuracy drop.
  • Outperformed DVT [46] in terms of accuracy-throughput trade-off.
  • Action Recognition:
    • Reduced computational cost of TimesFormer [4] by up to 33% on Kinetics 400 with a 0.5% loss in accuracy.
  • Achieved a 51% speedup over the MotionFormer baseline with a 0.1% accuracy increase.
  • Training Cost: ReViT's training cost is comparable to a standard ViT, significantly cheaper than methods like DVT that require multiple cascade models.
  • Inference Memory: The memory cost for inference is only slightly higher than baselines, making it practical.

Limitations and Future Work

  • Predefined Token Lengths: The current approach relies on a predefined set of token lengths. While efficient, it's not fully continuous.
  • Accuracy Drop with Very Small Tokens: The paper notes a significant accuracy drop when using very small token lengths (e.g., 4x4), which led them to exclude it from their predefined set.
  • Future Work: The authors believe their method is general and can be applied to other ViT architectures and tasks. Further exploration into more continuous token length adaptation or more sophisticated TLA designs could be future directions.

Summary updated

Abstract

The vision transformer is a model that breaks down each image into a sequence of tokens with a fixed length and processes them similarly to words in natural language processing. Although increasing the number of tokens typically results in better performance, it also leads to a considerable increase in computational cost. Motivated by the saying “A picture is worth a thousand words,” we propose an innovative approach to accelerate the ViT model by shortening long images. Specifically, we introduce a method for adaptively assigning token length for each image at test time to accelerate inference speed. First, we train a Resizable-ViT (ReViT) model capable of processing input with diverse token lengths. Next, we extract token-length labels from ReViT that indicate the minimum number of tokens required to achieve accurate predictions. We then use these labels to train a lightweight Token-Length Assigner (TLA) that allocates the optimal token length for each image during inference. The TLA enables ReViT to process images with the minimum sufficient number of tokens, reducing token numbers in the ViT model and improving inference speed. Our approach is general and compatible with modern vision transformer architectures, significantly reducing computational costs. We verified the effectiveness of our methods on multiple representative ViT models on image classification and action recognition.
BibTeX
@inproceedings{zhou2023make,
  author = {Qiqi Zhou and Yichen Zhu},
  title = {{Make a Long Image Short: Adaptive Token Length for Vision Transformers}},
  booktitle = {Ecml/pkdd},
  year = {2023},
  doi = {10.1007/978-3-031-43415-0\_5},
}
Figure