TL;DR
This paper introduces SwiftPrune, a novel Hessian-free weight pruning method that dramatically speeds up the process of compressing large language models (LLMs) without sacrificing accuracy. By avoiding computationally intensive Hessian matrix calculations and employing an efficient EWMA-based weight selection, SwiftPrune achieves significant speedups (up to 56x) and enables real-time pruning for LLMs.
What Problem Does This Paper Address?
Large Language Models (LLMs) have become incredibly powerful but also massive, leading to significant computational costs for deployment and inference. Model compression techniques, particularly pruning (removing redundant weights), are crucial for making these models more accessible. However, existing state-of-the-art pruning methods often rely on Hessian matrix calculations, which are computationally prohibitive for LLMs due to their quadratic scaling with parameter count. This makes them too slow for practical, real-time applications that might require frequent pruning. The paper aims to bridge this gap by developing a pruning method that is both computationally efficient and effective for LLMs.
Methods
SwiftPrune introduces two key innovations to overcome the limitations of Hessian-based pruning:
- Hessian-Free Weight Metric: Instead of relying on computationally expensive second-order derivatives (Hessian matrix), SwiftPrune uses a "contribution-based weight metric." This metric, derived from a simplified loss function approximation, focuses on the relative importance of weights rather than their absolute values, avoiding the need for Hessian computations.
- EWMA for Weight Selection: To further accelerate the process, SwiftPrune replaces traditional weight sorting with the Exponentially Weighted Moving Average (EWMA) technique. EWMA, borrowed from network communication protocols, efficiently estimates the mean and variance of weight contributions, allowing for rapid identification of weights to prune without explicit sorting.
The method is also extended to support structured sparsity pruning, which is more hardware-friendly and can lead to actual inference speedups on modern accelerators.
Datasets
The paper evaluates SwiftPrune on three prominent LLM families: Pythia, LLaMA2, and LLaMA3. For evaluating the performance of the pruned models, they use:
- Zero-shot perplexity (PPL): On WikiText2.
- Task-agnostic benchmarks: Including OpenCompass and the lm-evaluation-harness framework, which encompasses datasets like ARC (Easy and Challenge), WinoGrande, PIQA, HellaSwag, and OpenbookQA.
No new datasets were introduced.
Key Insights and Ideas
- Relative Importance Over Absolute Values: The core insight is that for post-training pruning, the relative importance of a weight (how much its removal impacts the loss) is more critical than its absolute magnitude or complex second-order derivative.
- EWMA as a Sorting Alternative: Using EWMA to track weight importance metrics (like estimated mean and L1-mean norm error) is a clever way to avoid the O(n log n) complexity of sorting, reducing it to O(n) for the pruning step.
- Structured Sparsity for Real Speedups: The extension to structured sparsity (specifically 2:4) is crucial because it aligns with hardware capabilities (like NVIDIA Tensor Cores), leading to actual inference latency reductions, not just parameter reduction.
- Pruning Without Retraining: A significant advantage is that SwiftPrune achieves its results without requiring any retraining or weight updates, making it practical for scenarios where models are frequently updated or adapted.
Key Results
SwiftPrune demonstrates remarkable performance improvements:
- Speedup: Achieves an average speedup of 12.29x (up to 56.02x) over existing state-of-the-art (SOTA) pruning methods. The pruning process completes within seconds.
- Accuracy: Maintains comparable accuracy, with an average performance decline of less than 2 percentage points compared to the original dense model when reaching 50% sparsity.
- Structured Sparsity Performance: For structured 2:4 sparsity, SwiftPrune achieves a 1.48x mean speedup in end-to-end inference latency and reduces DRAM access conflicts by 41%.
- Memory Efficiency: While not the primary focus, the paper notes SwiftPrune's memory usage is competitive (e.g., 20.01 GB for LLaMA2-7B vs. 22.79 GB for Wanda and 30.82 GB for SparseGPT).
Limitations and Future Work
- Memory Optimization: The authors acknowledge that while SwiftPrune is memory-efficient, further optimization is needed to enable deployment on more resource-constrained devices. This is identified as a critical focus for future research.
- Bias Preservation: The study also briefly touches on ethical concerns, noting that their approach preserves model performance while preventing bias amplification or the introduction of new discriminatory patterns, suggesting this is an area they've considered.
Summary updated