Shaping Weight Distributions for Robust Model Pruning

14 concepts8 papers

Why can some neural networks be stripped of 90% or more of their parameters with virtually no loss in performance, while others collapse immediately? A growing body of research shows that a model's resilience to pruning is not an accidental byproduct of size, but a direct consequence of its internal weight distribution.

When a network's weights are highly centralized—meaning the weight statistics are concentrated around a central point—it serves as a clear signal of overparameterization and redundancy, as noted by Duan et al. in The Crossword Puzzle: Simplifying Deep Neural Network Pruning with Fabulous Coordinates. This centralized state creates an ideal environment for weight removal. Conversely, deviations from this well-behaved distribution can severely hinder the pruning process.

Heavy Tails and Kurtosis as Pruning Indicators

To understand why a centralized distribution enables successful pruning, we must look at its mathematical shape, specifically its kurtosis (peakedness) and tail behavior.

As demonstrated by Sefidgaran et al. in Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks, the presence of heavy-tailed distributions in neural network weights means that the total representational signal is concentrated within a very small number of high-magnitude entries. Magnitude pruning exploits this exact property: it removes the large population of small-magnitude weights that contribute minimally to the total norm. As the network width increases, the error introduced by removing these small weights completely vanishes because the core signal remains safely isolated in the tails.

Mason-Williams et al. formalize this geometric intuition in What Makes a Good Prune? Maximal Unstructured Pruning for Maximal Cosine Similarity, showing that the kurtosis of a weight distribution is the primary indicator of its capacity for unstructured pruning. High-kurtosis distributions feature a sharp peak of near-zero weights and a few dominant, large weights. Because these few large weights dictate the overall representation, thousands of small weights can be safely eliminated without significantly altering the model's functional direction or reducing the cosine similarity between the parent and pruned network.

The Vulnerability of Wide and Dense Distributions

What happens when a model lacks this heavy-tailed, high-kurtosis structure? If a model has a wider weight distribution, removing connections eliminates significant representational information rather than minor noise. Gunes et al. explore this vulnerability in The Effect of Model Compression on Fairness in Facial Expression Recognition, noting that models with wider distributions experience higher information loss during sparsification, which can inadvertently widen accuracy gaps between different demographic groups.

Similarly, certain architectures exhibit internal parameter importance distributions that are highly dense. As Chen et al. show in P² Law: Scaling Law for Post-Training After Model Pruning, densely packed importance scores obscure the boundary between vital and redundant components. When pruning algorithms cannot cleanly distinguish between what is essential and what is excess, the resulting weight removal causes unpredictable and disproportionate damage to the model's underlying capabilities, making width pruning particularly difficult.

Actively Reshaping the Parameter Space

Because prunability depends so heavily on the shape of these distributions, researchers have shifted from passively observing weight shapes to actively optimizing them before, during, and after pruning.

Training-Time Regularization

One approach is to enforce a heavy-tailed structure during the initial training phase. In Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity, Yun et al. introduce a regularization mechanism that penalizes the reciprocal variance of absolute weight magnitudes. This forces the network to concentrate its knowledge into a small subset of high-magnitude weights while keeping the majority near zero, enabling extreme one-shot pruning of up to 96% of parameters without requiring expensive iterative retraining.

Pre-Pruning Rotations

Alternatively, the distribution can be reshaped right before pruning occurs. The authors of DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration (Gu et al.) use learnable orthogonal rotations to transform the weight distribution into a more concentrated state. By isolating noise and concentrating the signal within the parameter space, they demonstrate that a model's prunability is a dynamic property that can be actively maximized to reduce performance loss.

Iterative Re-densification

Even during the pruning process, the distribution remains the critical lever for performance. Li et al. propose the Sparse-Dense-Sparse (SDS) framework in Enhancing One-Shot Pruned Generative Pre-training Language Models through Sparse-Dense-Sparse Mechanism, which strategically re-densifies and then re-prunes models. This process optimizes the post-pruning weight distribution to be more "pruning-friendly," allowing for superior performance recovery and minimizing the flaws of standard pruning techniques that leave distributions unoptimized.

Preserving Kurtosis via Multi-Stage Pruning

Finally, for complex architectures like Mixture-of-Experts (MoE), the sequence of pruning matters. In STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning, Lee et al. show that pruning entire structural components or experts first helps maintain the kurtosis of the model's weight distribution. By preserving this peakedness early on, the model retains its structural robustness and can better tolerate subsequent fine-grained, unstructured pruning steps that would otherwise flatten the distribution and induce fragility.

Key Foundational Work

Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks (2021) — contributed 2 concept elements to this direction.
The Effect of Model Compression on Fairness in Facial Expression Recognition (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_architectural_prunability_f07d0576,
  title = {architectural prunability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-prunability/rdr_f07d05762eb745a9883e8bf20960291a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_architectural_prunability_f07d0576_v1,
  title = {architectural prunability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-prunability/rdr_f07d05762eb745a9883e8bf20960291a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake