oViT: An Accurate Second-Order Pruning Framework for Vision Transformers

Denis Kuznedelev, Eldar Kurtic, Elias Frantar, Dan Alistarh

Abstract

Modern vision architectures like Vision Transformers (ViTs) and ConvNext have proven significantly more difficult to prune than traditional CNNs due to complex inter-weight dependencies. We present Correlation-Aware Pruning (CAP), a second-order pruning framework that breaks this compressibility barrier. CAP utilizes a block-diagonal empirical Fisher matrix to capture weight correlations and employs a novel fast approximate solver to update saliency dynamically. Combined with a cyclic linear learning rate schedule, CAP achieves 75-80% unstructured sparsity on DeiT with less than a 1% drop in accuracy, leading to practical CPU speedups of up to 2.4x.

Introduction

While classic convolutional neural networks like ResNet50 are famously easy to prune—often retaining accuracy even at 90% sparsity—next-generation models like ViTs and ConvNext have proven much more stubborn. Previous state-of-the-art techniques struggled to push these models past 50% sparsity without significant performance degradation. The authors identify two main culprits: the complex inter-weight dependencies that lead to "collateral damage" during independent weight removal, and the notoriously difficult fine-tuning process required to recover accuracy after pruning.

Methodology

CAP addresses these challenges with two primary technical innovations:

  1. Efficient Correlation-Aware Pruning: Based on the Optimal Brain Surgeon (OBS) framework, CAP moves beyond simple diagonal approximations. It uses a block-diagonal empirical Fisher matrix to capture correlations. Crucially, it employs a novel solver that updates the importance (saliency) of remaining weights one-by-one as weights are removed. This dynamic adjustment prevents the accuracy "cliff" seen in static methods.
Comparison of pruning frameworks: CAP accounts for inter-weight correlations by updating saliency dynamically during pruning.
  1. Cyclic Fine-tuning Schedule: To help the model recover, the authors propose a cyclic linear learning rate schedule. By spiking the learning rate immediately after a pruning step and then decaying it, the model gets a "jump-start" to escape the sub-optimal state induced by pruning before settling into a new sparse configuration.

Key Results

Evaluations on ImageNet-1K demonstrate that CAP consistently outperforms other top-tier pruners like WoodFisher and M-FAC, especially at high sparsity levels. On the DeiT-Tiny architecture, CAP maintained high validation accuracy even as sparsity approached 80%, whereas competing methods saw performance collapse.

Validation accuracy vs. sparsity on DeiT-Tiny, comparing CAP against state-of-the-art pruning methods.
  • High Sparsity, Low Loss: CAP achieved 75-80% unstructured sparsity on DeiT with < 1% relative accuracy drop.
  • Practical Speedups: Using the DeepSparse inference engine, the authors demonstrated real-world throughput increases of 1.5x to 2.4x on standard CPUs.
  • Scalability: The method successfully pruned a massive ConvNext-L model (pretrained via CLIP on LAION-2B) to 50% sparsity with only a 0.3% accuracy loss.

Conclusion and Future Work

CAP proves that the "lack of compressibility" in ViTs is not inherent to the architecture, but rather a failure of traditional pruning tools. By accounting for how weights work together, these models become much more flexible. While second-order methods involve higher computational overhead than simple magnitude pruning, the accuracy gains and practical speedups justify the cost. Future work could involve extending this correlation-aware approach to even more diverse tasks like object detection and large language models (LLMs).

Summary updated

BibTeX
@inproceedings{kuznedelev2023ovit,
  author = {Denis Kuznedelev and Eldar Kurtic and Elias Frantar and Dan Alistarh},
  title = {{oViT: An Accurate Second-Order Pruning Framework for Vision Transformers}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=zYWtq_HUCoi},
}
Figure