SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization

Chuanyang Zheng, Zheyang Li, Kai Zhang, Zhi Yang, Wenming Tan, Jun Xiao, Ye Ren, Shiliang PuNeurIPS 2022 · OpenReview

TL;DR

SAViT introduces a "structure-aware" pruning framework for Vision Transformers (ViTs) that accounts for the complex interactions between different model components. By optimizing for "joint importance" rather than pruning components in isolation, the authors achieve significant FLOPs reductions (up to 50-70%) with minimal accuracy loss—and in some cases, even performance gains.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have revolutionized computer vision, but they come with a heavy "computational tax." Their massive memory footprint and high FLOP (Floating Point Operations) count make them difficult to deploy on edge devices like smartphones or embedded cameras.

While pruning—the process of removing redundant parameters—is a standard technique for slimming down neural networks, it is harder to execute effectively in ViTs than in traditional Convolutional Neural Networks (CNNs). CNNs are largely homogeneous, relying mostly on convolutional layers. ViTs, however, are highly heterogeneous. They consist of multi-head self-attention (MSA) modules, hidden neurons in feed-forward networks (FFN), and embedding dimensions.

Prunable components in a Vision Transformer block

Previous pruning methods often treated these components as independent silos. The authors argue that this isolated approach is suboptimal because it ignores how these parts interact. For example, pruning an embedding dimension might drastically change how an attention head functions.

Methods: The Power of Joint Importance

The core innovation of SAViT (Structure-Aware Vision Transformer) is the concept of joint importance. Instead of asking, "How much does this specific neuron matter?" the authors ask, "How much does the combination of these pruned components affect the total loss?"

To compute this efficiently, the researchers employed several key techniques:

  1. Taylor-based Approximation: They used a second-order Taylor expansion of the loss function to estimate the impact of pruning. This involves the Hessian matrix, which mathematically captures the interactions between different parameters.
  2. Complexity Reduction: Calculating a full Hessian for millions of parameters is computationally impossible. The authors simplified this by partitioning the interactions into "intra-component" (within a head) and "inter-component" (between heads and neurons) and using Monte Carlo methods to approximate the values.
  3. Collaborative Optimization: They formulated pruning as an optimization problem: minimize the change in loss subject to a specific computational budget (e.g., "reduce FLOPs by 50%").
  4. Evolutionary Algorithm (EA): To find the best pruning ratios for every single component across the entire network, they employed an Evolutionary Algorithm. This allows the system to "evolve" the most efficient sub-network architecture.

Key Insights and Ideas

  • Interactions Matter: The breakthrough realization is that in heterogeneous networks like ViTs, cross-component interactions are highly significant. Accounting for these interactions leads to a more "balanced" pruned architecture compared to methods that ignore them.
  • The Embedding Redundancy: The authors found that the embedding component often contains the most redundancy. By pruning all components comprehensively, they unlocked a much larger design space for optimization than previous methods that focused on just one area.
Performance comparison of isolated vs. comprehensive pruning and their interaction matrix
  • Pruning as a Regularizer: Surprisingly, pruning sometimes improved accuracy. For instance, their pruned DeiT-Base model actually outperformed the unpruned baseline by 0.7% while using 50% fewer FLOPs. This suggests that collaborative pruning can act as a form of regularization, removing noise and helping the model generalize better.

Key Results

The authors evaluated SAViT on ImageNet-1k for classification and COCO 2017 for object detection. The results across different scales of models (Base, Small, Tiny) were highly impressive:

  • DeiT-Base on ImageNet: SAViT achieved 82.75% Top-1 accuracy with a 40% reduction in parameters and ~40% reduction in FLOPs. This beats the original baseline accuracy of 81.84%.
  • Extreme Compression: Even when pushing the model to a 70% FLOPs reduction, the DeiT-Base model only dropped 0.18% in accuracy compared to the baseline, significantly outperforming other state-of-the-art pruning methods like UVC.
  • Object Detection: On the COCO dataset, they showed that 70% of the FLOPs in a Faster R-CNN (with a Swin-T backbone) could be removed with only a negligible 0.3% drop in mAP (mean Average Precision).
  • Hardware Speedup: On an Nvidia V100 GPU, the pruned models demonstrated real-world inference speedups ranging from 1.55x to 2.05x.

Limitations and Future Work

While the paper does not explicitly list a "Limitations" section, a few points can be inferred from the methodology:

  • Search Overhead: Although the pruning process itself is fast (less than 2 epochs), utilizing an Evolutionary Algorithm requires a search phase that might become time-consuming for extremely large-scale models (like ViT-Huge).
  • Task Specificity: While SAViT shows great results on classification and detection, it remains to be seen how well these specific "joint importance" metrics translate to generative tasks or self-supervised pre-training.
  • Future Directions: The authors suggest that this framework can be seamlessly extended to even more complex architectures. By simply defining new "components" (like the "neck" of a detection network), the collaborative optimization can handle the interactions and prune the network efficiently.

Summary updated

Abstract

Vision Transformers (ViTs) yield impressive performance across various vision tasks. However, heavy computation and memory footprint make them inaccessible for edge devices. Previous works apply importance criteria determined independently by each individual component to prune ViTs. Considering that heterogeneous components in ViTs play distinct roles, these approaches lead to suboptimal performance. In this paper, we introduce joint importance, which integrates essential structural-aware interactions between components for the first time, to perform collaborative pruning. Based on the theoretical analysis, we construct a Taylor-based approximation to evaluate the joint importance. This guides pruning toward a more balanced reduction across all components. To further reduce the algorithm complexity, we incorporate the interactions into the optimization function under some mild assumptions. Moreover, the proposed method can be seamlessly applied to various tasks including object detection. Extensive experiments demonstrate the effectiveness of our method. Notably, the proposed approach outperforms the existing state-of-the-art approaches on ImageNet, increasing accuracy by 0.7% over the DeiT-Base baseline while saving 50% FLOPs. On COCO, we are the first to show that 70% FLOPs of FasterRCNN with ViT backbone can be removed with only 0.3% mAP drop. The code is available at https://github.com/hikvision-research/SAViT.
BibTeX
@inproceedings{zheng2022savit,
  author = {Chuanyang Zheng and Zheyang Li and Kai Zhang and Zhi Yang and Wenming Tan and Jun Xiao and Ye Ren and Shiliang Pu},
  title = {{SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=9d8fJiXg7a},
}
Figure