Vision Transformers (ViTs) have fundamentally shifted the landscape of computer vision, delivering state-of-the-art accuracy across tasks ranging from image recognition to dense pixel-wise prediction. However, this performance comes at a steep cost. Unlike Convolutional Neural Networks (CNNs) that process images locally using sliding windows, ViTs treat images as sequences of patches and compute global self-attention. This design introduces a quadratic computational ($O(N^2)$) and memory bottleneck relative to the number of tokens, alongside massive parameter footprints.
Deploying these heavy models to resource-constrained edge devices—such as smartphones, drones, and embedded sensors—requires bridging a massive efficiency gap. To solve this, researchers are attacking the problem from three complementary angles: reducing token redundancy, compressing network structures via pruning and quantization, and co-designing hardware-aware architectures.
Reducing Token Redundancy
A core intuition behind optimizing ViTs is that images contain vast amounts of spatial redundancy. In a standard ViT, a background patch of clear sky consumes the exact same computational budget as a foreground object. By dynamically reducing the number of tokens processed in deeper layers, we can drastically lower the computational workload.
Early token pruning methods permanently discarded uninformative patches. However, this risks losing critical context if a early layer misjudges a token's importance. To address this, Xuwei Xu et al. proposed a "token idling" strategy in No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling. Instead of permanently deleting patches, their framework allows idle tokens to be re-selected and integrated back into computation in later layers. This provides a self-correction mechanism that preserves model accuracy while saving up to 33% of computational costs.
Standard ViT: [Token 1] -> [Layer 1] -> [Layer 2] -> [Layer 3] -> Output
[Token 2] -> [Layer 1] -> [Layer 2] -> [Layer 3] -> Output
Token Pruning: [Token 1] -> [Layer 1] -> [Layer 2] -> [Layer 3] -> Output
[Token 2] -> [Layer 1] -> (Dropped)
Dynamic Idling: [Token 1] -> [Layer 1] -> [Layer 2] -> [Layer 3] -> Output
[Token 2] -> [Layer 1] -> (Idled) -> [Layer 3] -> Output
Alternatively, one can determine token importance by looking at what the final layers actually utilize. In Patch Slimming for Efficient Vision Transformers, Chao Xu et al. introduced a top-down approach. By identifying the exact features required by the final classification layer, the framework prunes redundant patches backward through the network, achieving a 45% reduction in FLOPs with negligible impact on accuracy.
Structural Pruning and Weight Multiplexing
Beyond dropping tokens, efficiency can be engineered by shrinking the network's underlying dimensions: its width (the size of embeddings and number of attention heads) and its depth (the number of transformer blocks).
Pruning these components in isolation often breaks the delicate internal representations of the model. To combat this, Chuanyang Zheng et al. developed SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization. Rather than evaluating components independently, SAViT optimizes for "joint importance," capturing the complex interactions between attention heads and linear layers to reduce FLOPs by up to 70%. Similarly, the framework proposed by Fang Yu et al. in Width & Depth Pruning for Vision Transformers uses learnable saliency scores alongside plug-in intermediate classifiers to slim down width and depth simultaneously.
Instead of removing layers entirely, we can also compress the model by forcing layers to share parameters. Houwen Peng et al. introduced this concept as "weight multiplexing" in MiniViT: Compressing Vision Transformers with Weight Multiplexing. The authors share weights across multiple transformer blocks but apply tiny, unique transformations to the shared weights in each layer to maintain representational diversity. Combined with multi-level knowledge distillation, this technique slashes parameter counts by up to 9.7×. To automate the search for these compact structures, Shixing Yu frames rank selection as a Neural Architecture Search (NAS) problem in FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer, using low-rank approximations to systematically find the optimal compression rank for every individual layer.
Overcoming the Quantization Bottleneck
Post-Training Quantization (PTQ) is the process of converting high-precision floating-point numbers (FP32) into low-bit integers (like INT8 or INT4) without retraining the model. While highly effective for CNNs, standard PTQ frequently breaks ViTs. The culprit lies in the non-linear operations: LayerNorm and Softmax activations exhibit extreme value distributions and high variance, which integer representations cannot easily capture.
To achieve a fully quantized ViT, Yang Lin developed FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer. The author introduced "Power-of-Two Factors" and a specialized "Log-Int-Softmax" mathematical approximation, allowing the model to perform attention calculations entirely on integer-only hardware without suffering severe accuracy drops.
Standard Quantization: [FP32 Values] --------> Linear Mapping --------> [Uniform INT8] (Distorts Outliers)
FQ-ViT Quantization: [Softmax/LN] --> Log-Integer / Power-of-Two --> [Log-Scale INT8] (Preserves Variance)
For ultra-low resource settings, researchers are pushing quantization to its logical extreme: 1-bit binarization. Sheng Xu et al. addressed the severe "attention distortion" caused by binarization in Bi-ViT: Pushing the Limit of Vision Transformer Quantization. By introducing learnable scaling factors and a ranking-aware distillation scheme, they achieved a theoretical 61.5× speedup, proving that even 1-bit ViTs can maintain strong performance.
Hardware-Aware Co-Design
Software optimizations yield the highest returns when tailored directly to the underlying execution hardware. Field-Programmable Gate Arrays (FPGAs) and custom application-specific circuits offer massive efficiency gains if the model's mathematical operations align with the hardware's physical layout.
In Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization, Zhengang Li 0001 et al. co-designed an automation pipeline that maps ViTs to FPGAs. The framework strategically assigns fixed-point precision to linear layers and power-of-two precision to non-linear layers, matching the hardware's native capabilities to achieve a 5.6× speedup over standard baselines.
Taking hardware customization a step further, Tong Xie et al. introduced ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer. ASCEND replaces power-hungry digital multipliers with Stochastic Computing (SC) circuits, which represent numbers as random bitstreams and perform multiplication using simple, single-gate AND circuits. By co-designing specialized hardware circuits for GELU and Softmax alongside a noise-aware training pipeline, they achieved a 12.6× reduction in the area-delay product.
Finally, hardware-aware design can exploit structural sparsity. Hanxue Liang et al. developed Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts. By routing different image tokens to specialized "expert" sub-networks on an FPGA, their architecture achieves real-time multi-task vision processing at a fraction of the energy cost of a traditional GPU.