Abstract
Large language models have demonstrated capabilities in text generation, while their increasing parameter scales present challenges in computational and memory efficiency. Post-training sparsity (PTS), which reduces model cost by removing weights from dense networks, is an effective approach. However, native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states, resulting in unsatisfactory performance recovery even with post-tuning. We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning, to push the limits of PTS. At the distribution level, we enhance distributional sparsity through mathematically equivalent scaling transformations, which are fully absorbable and incur no extra parameters or inference-time overhead. At the feature level, we introduce Spectral Norm Loss to promote feature sparsity from a low-rank perspective. Experiments across diverse model architectures and tasks demonstrate that our method further enhances sparsity-friendliness, achieving superior pruning performance over existing approaches.
Introduction
As Large Language Models (LLMs) scale to hundreds of billions of parameters, their deployment becomes prohibitively expensive in terms of both memory and compute. Post-Training Sparsity (PTS)—the process of removing weights from a pre-trained model—has emerged as a popular solution because it avoids the massive cost of retraining from scratch.
However, current pruning methods, such as SparseGPT or Wanda, have hit a performance plateau. These methods primarily focus on designing sophisticated "importance scores" to decide which weights to cut. The fundamental challenge is that standard LLMs are not naturally sparse; their weight distributions are dense and "long-tailed." When a dense model is forced into a sparse state, its internal representations are disrupted, leading to significant performance degradation. This research shifts the paradigm from how to prune to how to prepare the model for pruning through a framework called Sparsity Induction (SI).
Methodology
Sparsity Induction (SI) is a "training-light" framework that proactively reshapes a model’s weight distribution to make it more "sparsity-friendly" before any weights are actually removed.
1. Distributional Induction via Scaling Transformations
The core technical innovation involves per-channel scaling and shifting factors that are mathematically "absorbable." For any given linear layer, the authors reparameterize the weights and inputs such that the mathematical output remains identical, but the underlying weight distribution changes. By optimizing these scaling factors, the model is encouraged to "push" more mass into a smaller subset of weights, effectively sharpening the distribution.
Crucially, because these transformations are mathematically equivalent, the scaling factors can be folded back into the original weights after the induction phase. This results in zero inference-time overhead and no additional parameters.
2. Feature-Level Induction
To ensure model stability at high sparsity levels, the framework introduces a Spectral Norm Loss. This loss encourages weight matrices to maintain a low-rank structure. From a feature perspective, this makes the model's representations more redundant and compressible, ensuring that the removal of weights does not destroy critical information pathways.
3. Fast Hessian Update
Pruning often relies on second-order information (the Hessian matrix) to minimize reconstruction error. To make SI efficient, the researchers developed a "Fast Hessian" proxy. Instead of repeatedly calculating complex derivatives, they use a diagonal approximation that can be refreshed in constant time. This optimization results in a 22.65x speedup compared to classical recomputation methods, allowing the induction process to complete in seconds rather than minutes.
Experimental Results
The researchers evaluated SI using the OPT family (125M to 2.7B) and LLaMA-1/2 (7B and 13B) models. Calibration was performed using only 128 samples from the C4 dataset over 1–5 epochs.
Language Modeling Performance
On WikiText-2 and C4, SI consistently outperformed state-of-the-art pruning baselines. For example, on LLaMA-1 7B at 50% unstructured sparsity, SI improved the Wanda baseline's perplexity from 7.26 to 7.04. The gains were even more significant in semi-structured scenarios like 2:4 and 4:8 sparsity patterns.
Zero-Shot Generalization
Across six downstream benchmarks (including ARC, HellaSwag, and BoolQ), SI boosted average accuracy. For LLaMA-2 13B with 4:8 sparsity, SI improved SparseGPT’s performance from 55.09% to 55.10%, and Wanda’s from 54.59% to 54.79%.
Efficiency and Latency
Beyond accuracy, the method is highly practical. The "Fast Update" reduced the time to refresh pruning metrics on a LLaMA-7B from 345 seconds to just 15 seconds. Furthermore, in a 2:4 sparsity pattern, the SI-enhanced model achieved a 1.24x end-to-end speedup over the dense baseline during inference, proving that the method successfully translates theoretical sparsity into real-world performance gains.
Conclusion
Sparsity Induction demonstrates that a model’s "prunability" is a trainable trait rather than a fixed property. By investing a minimal amount of compute to reshape weight distributions before pruning, researchers can achieve the benefits of specialized sparse architectures without the need for custom inference engines. Future work may explore combining SI with quantization to further push the boundaries of LLM compression.
Summary updated