Traditional neural network pruning is often a "one-way street." In standard magnitude-based pruning, weights are evaluated against a fixed threshold; those that fall below it are permanently deleted. As Haipeng Jia and colleagues note in DropPruning for Model Compression, this "death sentence" for small weights is risky. A weight might be small during one phase of training but become vital later.
Probabilistic pruning replaces this rigid threshold with a stochastic gate. By treating the existence of a connection as a random variable, models can explore different sparse architectures during training, allowing "unimportant" weights to be recovered if they prove useful later.
Learning to Prune via Differentiable Masks
The core mechanism of this research direction is the stochastic mask. Instead of multiplying a weight $w$ by a deterministic binary value, we multiply it by a random variable $m$ sampled from a distribution (often Bernoulli). The probability $p$ of this mask being "on" is itself a trainable parameter.
Christos Louizos and Max Welling pioneered this by using $L_0$ regularization with a "hard concrete" distribution in Learning Sparse Neural Networks through L0 Regularization. Because the $L_0$ norm (the count of non-zero elements) is non-differentiable, they used a continuous relaxation that allows gradient descent to optimize the pruning probabilities alongside the weights.
Similarly, Chee Seng Chan and Joon Huang Chuah applied this to image captioning in End-to-End Supermask Pruning, demonstrating that learning a "supermask" through Bernoulli sampling can compress models by over 90% while maintaining performance.
Exploration and the Recovery of Weights
Why use randomness instead of just picking the largest weights? The intuition is rooted in exploration. Deterministic pruning is greedy; it assumes the current magnitude of a weight is a perfect proxy for its long-term importance.
Huan Wang and colleagues argue in Structured Probabilistic Pruning for Convolutional Neural Network Acceleration that stochasticity prevents the model from over-pruning based on a single snapshot of weight values. By using Monte Carlo sampling, the network can "try out" different sparse configurations. If a weight is pruned but the loss increases significantly, the gradient will push its probability $p$ back up, effectively "recovering" the connection.
This concept is taken to a physical extreme by Shahrokh Valaee and Alex Labach in A Framework for Neural Network Pruning Using Gibbs Distributions. They treat pruning as a phase transition in statistical physics, using an "annealing" strategy where the system starts highly stochastic and gradually settles into a fixed sparse structure as training progresses.
Scaling to Large Models and Hardware Constraints
As models grow into the billions of parameters, the memory cost of maintaining a probability for every weight becomes prohibitive. Recent work has focused on making probabilistic pruning efficient and hardware-aware.
- N:M Sparsity: Modern GPUs often require "semi-structured" sparsity (e.g., exactly 2 out of every 4 weights must be zero). Xikun Zhang and colleagues introduced MaskPro, which uses a categorical distribution over blocks of weights to satisfy these constraints without the exponential memory overhead of previous methods.
- Channel Pruning: Instead of individual weights, Wenqi Shao and Zhenguo Li focus on entire filters in BWCP. They use Gumbel-Softmax sampling to stochastically prune channels during training, allowing the model to learn which entire feature maps are redundant.
- Gradient Sparsity: Brian Chmiel and Itay Hubara extend this logic to the training process itself in Optimal Fine-Grained N: M sparsity for Activations and Neural Gradients. By stochastically selecting which gradients to keep, they accelerate the "update phase" of training while avoiding the systematic bias that deterministic pruning would introduce into the gradient direction.
Theoretical Foundations in Sketching and Inference
Beyond empirical success, the effectiveness of probabilistic pruning is increasingly supported by theoretical frameworks. Noga Bar and Raja Giryes frame pruning as a sketching problem—a randomized matrix approximation—in Pruning at Initialization - A Sketching Perspective. They show that randomized sampling preserves the network's expressivity and gradient flow more effectively than rigid "hard thresholding," especially when pruning before training begins.
From a Bayesian perspective, Panagiotis Dimitrakopoulos and Giorgos Sfikas demonstrate in Variational Feature Pyramid Networks that pruning can be viewed as learning a posterior distribution over connections. By starting with a dense "super-network" and using variational inference to identify redundant paths, the model effectively "discovers" the optimal architecture for a specific dataset.