Image Recognition with Online Lightweight Vision Transformer: A Survey

Zherui Zhang, Rongtao Xu, Jie Zhou, Changwei Wang, Xingtian Pei, Wenhao Xu, Jiguang Zhang, Li Guo, Longxiang Gao, Wenbo Xu, Shibiao XuarXiv 2025 · arXiv · OpenReview

Abstract

The Transformer architecture has achieved significant success in natural language processing, motivating its adaptation to computer vision tasks. Unlike convolutional neural networks, vision transformers inherently capture long-range dependencies and enable parallel processing, yet lack inductive biases and efficiency benefits, facing significant computational and memory challenges that limit its real-world applicability. This paper surveys various online strategies for generating lightweight vision transformers for image recognition, focusing on three key areas: Efficient Component Design, Dynamic Network, and Knowledge Distillation. We evaluate the relevant exploration for each topic on the ImageNet-1K benchmark, analyzing trade-offs among precision, parameters, throughput, and more to highlight their respective advantages, disadvantages, and flexibility. Finally, we propose future research directions and potential challenges in the lightweighting of vision transformers with the aim of inspiring further exploration and providing practical guidance for the community.

Introduction

Vision Transformers (ViTs) have emerged as powerful alternatives to Convolutional Neural Networks (CNNs), primarily due to their ability to model global context through self-attention mechanisms. However, the quadratic computational complexity relative to the number of input tokens and the lack of inherent inductive biases (such as translation invariance) pose significant hurdles for deployment on resource-constrained edge devices.

To address these challenges, researchers have developed various compression techniques. While "offline" methods like post-training pruning and quantization are common, this survey focuses on online lightweighting. As illustrated in Figure 1, online strategies optimize the model's architecture and training process itself, creating inherently efficient models that serve as a robust foundation for further offline compression.

Comparison of Online vs. Offline Lightweighting

Taxonomy of Online Lightweighting

The landscape of online lightweighting for ViTs can be categorized into three primary pillars: Efficient Component Design, Dynamic Networks, and Knowledge Distillation. This taxonomy provides a structured roadmap for understanding how different methods target specific bottlenecks in the ViT pipeline.

Taxonomy of Online Lightweighting Strategies

1. Efficient Component Design

This pillar focuses on re-engineering the fundamental building blocks of the Transformer to reduce complexity and re-introduce beneficial inductive biases.

  • Embedding and Position: Instead of simple linear projections, methods like Dynamic Patch Partitioning (DPP) and Local Patch Embedding (LPE) utilize CNN-like structures to capture fine-grained local details.
  • Token Updates: The standard self-attention mechanism is often replaced with more efficient variants. Partial Attention (e.g., Swin’s shifted windows) restricts attention to local neighborhoods, while Linear Attention and Attention-Free architectures (like MLP-Mixer) aim to eliminate the $O(N^2)$ bottleneck entirely.
  • Framework Design: Hierarchical frameworks, which gradually reduce spatial resolution while increasing feature depth, have proven highly effective for vision tasks.

2. Dynamic Networks

Dynamic networks adapt their computational load based on the complexity of the input data, allowing for significant savings on "easy" samples.

  • Dynamic Resolution: Techniques such as Token Pruning discard uninformative background patches, while Token Merging combines similar patches to reduce the workload in deeper layers.
  • Depth Adaptation: Early Exit strategies allow the model to output a classification at shallower layers if the confidence is high enough, bypassing unnecessary deeper computations.

3. Knowledge Distillation (KD)

Knowledge Distillation involves training a compact "student" model to mimic the behavior of a high-performance "teacher" model.

  • Feature-Level Distillation: The student is trained to match the intermediate feature maps or attention maps of the teacher, capturing its internal representation logic.
  • Response-Level Distillation: The student learns from the "soft labels" (probability distributions) produced by the teacher, which contain more information about inter-class relationships than standard one-hot labels.

Performance Benchmarks

The survey evaluates these methods primarily on the ImageNet-1K benchmark, focusing on the trade-off between accuracy and efficiency. Key findings include:

  • State-of-the-Art Efficiency: Models like SSA-B achieve a high 84.0% Top-1 accuracy with only 39.6M parameters.
  • Extreme Lightweighting: Evo-ViT represents the extreme end of efficiency, maintaining high throughput with a mere 5.9M parameters.
  • Dynamic Gains: DynamicViT demonstrates that pruning up to 70% of tokens in deeper layers can increase inference throughput by 36.6% with a negligible accuracy drop of 0.5%.
  • Peak Performance: Hierarchical designs like Hiera-H currently lead in raw performance, reaching 86.9% Top-1 accuracy.

Future Directions and Conclusion

While significant progress has been made, several challenges remain for the community:

  • Automation: Moving away from manual layer selection toward Automatic Configuration and end-to-end automated pipelines.
  • Spatial Integrity: Developing better methods to preserve spatial relationships after aggressive token pruning or merging.
  • Privacy and Security: Exploring Data-Free Knowledge Distillation (DFKD) for privacy-sensitive applications and improving the adversarial robustness of lightweight models.

In conclusion, online lightweighting strategies are essential for bridging the gap between the theoretical power of Vision Transformers and their practical deployment in real-world, resource-constrained environments.

Summary updated

Abstract

The Transformer architecture has achieved significant success in natural language processing, motivating its adaptation to computer vision tasks. Unlike convolutional neural networks, vision transformers inherently capture long-range dependencies and enable parallel processing, yet lack inductive biases and efficiency benefits, facing significant computational and memory challenges that limit its real-world applicability. This paper surveys various online strategies for generating lightweight vision transformers for image recognition, focusing on three key areas: Efficient Component Design, Dynamic Network, and Knowledge Distillation. We evaluate the relevant exploration for each topic on the ImageNet-1K benchmark, analyzing trade-offs among precision, parameters, throughput, and more to highlight their respective advantages, disadvantages, and flexibility. Finally, we propose future research directions and potential challenges in the lightweighting of vision transformers with the aim of inspiring further exploration and providing practical guidance for the community. Project Page: https://github.com/ajxklo/Lightweight-VIT
BibTeX
@misc{zhang2025image,
  author = {Zherui Zhang and Rongtao Xu and Jie Zhou and Changwei Wang and Xingtian Pei and Wenhao Xu and Jiguang Zhang and Li Guo and Longxiang Gao and Wenbo Xu and Shibiao Xu},
  title = {{Image Recognition with Online Lightweight Vision Transformer: A Survey}},
  year = {2025},
  eprint = {2505.03113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.03113},
}
Figure