Abstract
Modern vision architectures like Vision Transformers (ViTs) and ConvNext have proven significantly more difficult to prune than traditional CNNs due to complex inter-weight dependencies. We present Correlation-Aware Pruning (CAP), a second-order pruning framework that breaks this compressibility barrier. CAP utilizes a block-diagonal empirical Fisher matrix to capture weight correlations and employs a novel fast approximate solver to update saliency dynamically. Combined with a cyclic linear learning rate schedule, CAP achieves 75-80% unstructured sparsity on DeiT with less than a 1% drop in accuracy, leading to practical CPU speedups of up to 2.4x.
Introduction
While classic convolutional neural networks like ResNet50 are famously easy to prune—often retaining accuracy even at 90% sparsity—next-generation models like ViTs and ConvNext have proven much more stubborn. Previous state-of-the-art techniques struggled to push these models past 50% sparsity without significant performance degradation. The authors identify two main culprits: the complex inter-weight dependencies that lead to "collateral damage" during independent weight removal, and the notoriously difficult fine-tuning process required to recover accuracy after pruning.
Methodology
CAP addresses these challenges with two primary technical innovations:
- Efficient Correlation-Aware Pruning: Based on the Optimal Brain Surgeon (OBS) framework, CAP moves beyond simple diagonal approximations. It uses a block-diagonal empirical Fisher matrix to capture correlations. Crucially, it employs a novel solver that updates the importance (saliency) of remaining weights one-by-one as weights are removed. This dynamic adjustment prevents the accuracy "cliff" seen in static methods.
- Cyclic Fine-tuning Schedule: To help the model recover, the authors propose a cyclic linear learning rate schedule. By spiking the learning rate immediately after a pruning step and then decaying it, the model gets a "jump-start" to escape the sub-optimal state induced by pruning before settling into a new sparse configuration.
Key Results
Evaluations on ImageNet-1K demonstrate that CAP consistently outperforms other top-tier pruners like WoodFisher and M-FAC, especially at high sparsity levels. On the DeiT-Tiny architecture, CAP maintained high validation accuracy even as sparsity approached 80%, whereas competing methods saw performance collapse.
- High Sparsity, Low Loss: CAP achieved 75-80% unstructured sparsity on DeiT with < 1% relative accuracy drop.
- Practical Speedups: Using the DeepSparse inference engine, the authors demonstrated real-world throughput increases of 1.5x to 2.4x on standard CPUs.
- Scalability: The method successfully pruned a massive ConvNext-L model (pretrained via CLIP on LAION-2B) to 50% sparsity with only a 0.3% accuracy loss.
Conclusion and Future Work
CAP proves that the "lack of compressibility" in ViTs is not inherent to the architecture, but rather a failure of traditional pruning tools. By accounting for how weights work together, these models become much more flexible. While second-order methods involve higher computational overhead than simple magnitude pruning, the accuracy gains and practical speedups justify the cost. Future work could involve extending this correlation-aware approach to even more diverse tasks like object detection and large language models (LLMs).
Summary updated