Abstract
Pruning large-scale models like Llama-7B or Segment Anything (SAM) is essential for deployment on consumer hardware but often results in significant performance degradation at high sparsity levels. This "performance cliff" is primarily caused by error accumulation in one-shot pruning methods. We present Immediate Compensation Pruning (ICP), a strategy that bridges the gap between zero-shot pruning and full fine-tuning. ICP utilizes a sliding-window mechanism to compensate for errors block-by-block and intelligently rearranges sparsity targets across the network. Our results show that ICP maintains high performance at 50–70% sparsity while remaining memory-efficient, outperforming state-of-the-art methods like SparseGPT and Wanda.
The Challenge: The Error Propagation Cliff
Existing pruning methods generally fall into two categories: iterative pruning and one-shot pruning. Iterative pruning (prune-and-fine-tune) yields high performance but requires massive GPU memory for gradients and optimizer states—often exceeding the original model's requirements. One-shot methods (e.g., SparseGPT, Wanda) are fast and memory-efficient but suffer from error accumulation.
At low sparsity (10–30%), one-shot methods are effective. However, at "mid-to-high" sparsity (50–70%), small errors introduced in early layers snowball as they propagate through the network. By the time the signal reaches the final layer, the model's output is often degraded. This error propagation is the primary barrier to high-sparsity compression on a budget.
Methodology: Stopping the Snowball
The authors propose two main innovations to prevent the error snowball:
1. Block-wise Compensate Pruning (The Sliding Window)
Instead of pruning the entire model simultaneously, ICP employs a sliding window of two blocks ($B^i$ and $B^{i+1}$). First, it prunes block $B^i$ using standard one-shot techniques. Immediately after, it uses the unpruned block $B^{i+1}$ to compensate for the errors introduced by $B^i$. It treats $B^{i+1}$ as a small trainable module, adjusting its weights to match the original, unbiased output of the previous block. The window then slides forward: $B^{i+1}$ (now compensated) is pruned, and $B^{i+2}$ fixes its errors. This "immediate" fix prevents errors from propagating deep into the backbone while keeping memory usage low, as only one or two blocks are ever active in memory.
2. Sparsity Rearrangement
ICP also introduces strategic sparsity redistribution to optimize performance:
- Inter-Block: The final block of a model has no successor to absorb its errors. ICP shifts some pruning burden away from the last block to earlier blocks that can be compensated.
- Intra-Block: Within a Transformer block, some layers are more sensitive than others. ICP reduces sparsity in the critical QKV (Attention) matrices and increases it in the easier-to-compensate FC2 (Feed-Forward) layers.
Evaluation and Results
ICP was evaluated across language (OPT, Llama 2-7B) and vision (SAM) domains, consistently outperforming Magnitude, SparseGPT, and Wanda as sparsity increased.
- Language Performance: On OPT-1.3B at 70% sparsity, ICP achieved a perplexity of 36.78, significantly better than SparseGPT (51.75) and Wanda (99.53). On Llama 2-7B, ICP maintained a mean zero-shot accuracy of 46.94%, compared to SparseGPT's 41.79%.
- Vision Performance: For SAM-H at an extreme 90% sparsity, ICP maintained a respectable 57.60% IoU, while Wanda and SparseGPT collapsed to 1.20% and 18.15%, respectively.
- Efficiency: ICP is highly memory-efficient. It pruned OPT-6.7B using only 7.8 GB of VRAM, allowing a 7B parameter model to be processed on a single consumer GPU (e.g., RTX 3090) without exceeding memory limits.
Conclusion
ICP provides a robust solution for high-sparsity model compression without the prohibitive costs of full fine-tuning. By combining block-wise compensation with intelligent sparsity rearrangement, it overcomes the error propagation bottleneck, enabling the deployment of large-scale models on resource-constrained hardware.
Paper Citation: Luo, X., Fu, X., Jiang, Z., & Zhou, S. K. (2024). ICP: Immediate Compensation Pruning for Mid-to-high Sparsity. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Summary updated