SwiftPrune: Hessian-Free Weight Pruning for Large Language Models

Yuhan Kang, Yang Shi, Mei We, Jun He, Jianchao Yang, Zeyu Xue, Jing Feng, Xinwang Liu

Abstract

Post-training pruning is essential for compressing large language models (LLMs) for efficient deployment. However, state-of-the-art methods rely on computationally expensive Hessian matrix calculations, limiting their speed and scalability. We introduce SwiftPrune, a Hessian-free weight pruning method that achieves significant speedups through two key innovations: a contribution-based weight metric that avoids second-order derivatives and an Exponentially Weighted Moving Average (EWMA) technique to bypass weight sorting. SwiftPrune prunes models in seconds, achieving up to a 56x speedup over existing methods while maintaining comparable accuracy and supporting structured sparsity for hardware acceleration.

Introduction

As LLMs grow exponentially in size, compressing them for practical deployment has become critical. Post-training pruning—removing redundant weights from a fully trained model—is a popular solution. However, current state-of-the-art methods like SparseGPT and Wanda rely on second-order derivatives (the Hessian matrix) to identify weights safe for deletion. Calculating the Hessian scales poorly ($O(n^3)$), making it a bottleneck for rapid iteration. SwiftPrune addresses this by identifying unimportant weights in linear time ($O(n)$).

Methods

SwiftPrune's efficiency stems from two primary algorithmic shifts:

  1. Hessian-Free Contribution Metric: Instead of calculating the exact impact of removing a weight via the Hessian, SwiftPrune uses a mathematical approximation. In high-dimensional LLM layers, certain terms in the Taylor expansion dominate. By treating the inverse Hessian as a constant, SwiftPrune reduces scoring complexity from $O(n^3)$ to $O(n)$.
  2. EWMA for Thresholding (No Sorting): Traditional pruning requires sorting weights ($O(n \log n)$) to find a threshold. SwiftPrune eliminates sorting by using Exponentially Weighted Moving Average (EWMA), a technique borrowed from network protocols. It treats weights as a sequence, updating a running mean and variance to prune weights on the fly if they fall below a dynamic threshold.
The SwiftPrune workflow, illustrating the Hessian-free contribution metric and the EWMA-based state update for pruning without sorting.
  1. Structured Sparsity Support: The method is extended to support 2:4 structured sparsity, a pattern natively accelerated by NVIDIA's Ampere Tensor Cores.

Key Insights

  • Relative Importance Suffices: Pruning doesn't require exact loss contributions; relative rankings are enough to identify redundant weights. By focusing on the scale disparity in wide LLM layers, aggressive algebraic approximations introduce negligible error.
  • Cross-Disciplinary Optimization: Repurposing a network routing algorithm (TCP's EWMA) to avoid sorting arrays is a novel optimization that drastically cuts computational overhead.
Jitter variation plots showing how the EWMA-estimated mean (est) and deviation (dev) closely track real values in the MLP layers, validating the thresholding approach.

Results

SwiftPrune was evaluated on Pythia (2.8B), LLaMA2 (7B, 13B), and LLaMA3.1 (8B) across standard benchmarks like WikiText2 and OpenCompass.

  • Blazing Speed: SwiftPrune completes the pruning process in seconds. It achieves an average speedup of 12.29x (and up to 56.02x) over existing SOTA approaches. For example, pruning LLaMA2-7B takes just 7.73 seconds, compared to Wanda (114s) and SparseGPT (410s).
  • Maintained Accuracy: At 50% unstructured sparsity, SwiftPrune matches the accuracy of much slower Hessian-based methods within 2 percentage points. In 2:4 hardware-friendly sparsity scenarios, SwiftPrune actually outperforms Wanda and SparseGPT, achieving 3.7% to 12.7% higher accuracy.
  • Hardware Acceleration: The 2:4 structured sparsity translates to a 1.48x mean speedup in end-to-end inference latency on an RTX 4090 GPU, while reducing DRAM access conflicts by 41%.

Conclusion

SwiftPrune demonstrates that Hessian-free pruning is not only possible but superior for high-speed LLM compression. By leveraging scale and efficient estimation algorithms, it enables rapid model optimization without sacrificing accuracy. Future work will focus on further optimizing memory usage to allow pruning on resource-constrained edge devices.

Summary updated

Abstract

Post-training pruning, as one of the key techniques for compressing large language models, plays a vital role in lightweight model deployment and model sparsity. However, current mainstream pruning methods dependent on the Hessian matrix face significant limitations in both pruning speed and practical effectiveness due to the computationally intensive nature of second-order derivative calculations. This paper presents SwiftPrune, a novel Hessian-free weight pruning method that achieves hardware-efficient model compression through two key innovations: 1) SwiftPrune eliminates the need for computationally intensive Hessian matrix calculations by introducing a contribution-based weight metric, which evaluates the importance of weights without relying on second-order derivatives. 2) we employ the Exponentially Weighted Moving Average (EWMA) technique to bypass weight sorting, enabling the selection of weights that contribute most to LLM accuracy and further reducing time complexity. Our approach is extended to support structured sparsity pruning, facilitating efficient execution on modern hardware accelerators. We validate the SwiftPrune on three LLMs (namely LLaMA2, LLaMA3, and Pythia), demonstrating that it significantly enhances compression performance. The experimental findings reveal that SwiftPrune completes the pruning process within seconds, achieving an average speedup of 12.29x (up to 56.02x) over existing SOTA approaches.
BibTeX
@inproceedings{kang2025swiftprune,
  author = {Yuhan Kang and Yang Shi and Mei We and Jun He and Jianchao Yang and Zeyu Xue and Jing Feng and Xinwang Liu},
  title = {{SwiftPrune: Hessian-Free Weight Pruning for Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure