Abstract
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural network, it is possible to find small sub-networks which, when trained from scratch on a comparable budget, match the performance of the original dense counterpart. We revisit fundamental aspects of pruning algorithms, pointing out missing ingredients in previous approaches, and develop a method, Continuous Sparsification, which searches for sparse networks based on a novel approximation of an intractable $\ell_0$ regularization. We compare against dominant heuristic-based methods on pruning as well as ticket search -- finding sparse subnetworks that can be successfully re-trained from an early iterate. Empirical results show that we surpass the state-of-the-art for both objectives, across models and datasets, including VGG trained on CIFAR-10 and ResNet-50 trained on ImageNet. In addition to setting a new standard for pruning, Continuous Sparsification also offers fast parallel ticket search, opening doors to new applications of the Lottery Ticket Hypothesis.
Introduction
Deep neural networks are notoriously overparameterized, leading to massive memory footprints and slow inference times. The Lottery Ticket Hypothesis (LTH) suggests that within these giant networks, there exist tiny "winning tickets": sparse subnetworks that, if trained from scratch with the right initialization, can perform just as well as the full model. Historically, the gold standard for finding these tickets has been Iterative Magnitude Pruning (IMP), a heuristic that iteratively trains, prunes small weights, and rewinds. While effective, IMP is computationally expensive and lacks a strong theoretical foundation.
Researchers have developed Continuous Sparsification (CS), a new method that turns the "black art" of pruning into a principled optimization problem. By using a deterministic approximation of $\ell_0$ regularization, CS discovers winning tickets faster and more effectively than previous state-of-the-art heuristics.
Methodology: Continuous Sparsification
CS frames pruning as a loss minimization problem using a deterministic mask, avoiding the noise issues that plagued previous stochastic $\ell_0$ methods.
- The Masking Function: Instead of binary 0/1 masks, CS uses a sigmoid-based soft mask: $m = \sigma(\beta s)$, where $s$ is a learnable parameter and $\beta$ is a "temperature" hyperparameter.
- Temperature Annealing: Training begins with a low $\beta$ (a smooth, soft mask) which is gradually increased. As $\beta \to \infty$, the sigmoid converges to a hard Heaviside step function, creating a smooth continuation path from a dense to a sparse network.
- Deterministic Optimization: By remaining entirely deterministic, the optimizer receives stable gradients, allowing standard optimizers like SGD to converge more effectively than stochastic alternatives.
- No Rewinding Required: Unlike IMP, CS can find excellent subnetworks even when maintaining weights between rounds, simplifying the search process.
Experimental Results
State-of-the-Art Pruning
The method was rigorously tested on standard benchmarks including CIFAR-10 (VGG-16, ResNet-20) and ImageNet (ResNet-50). On VGG-16, CS achieved 99.6% sparsity while staying within 2% of the original model's accuracy, significantly outperforming AMC, GMP, and Network Slimming.
On the large-scale ImageNet task, CS consistently beat competitive methods like STR and DNW. For instance, at ~95% sparsity on ResNet-50, CS achieved 72.4% Top-1 accuracy, compared to 69.2% for IMP.
Winning Ticket Search
CS successfully identified "winning tickets" that, when re-trained from an early iterate, surpassed those found by IMP in both sparsity and final accuracy. Furthermore, CS allows for fast parallel ticket search. By varying initial mask values, researchers can run multiple pruning experiments in parallel to find different sparsity levels simultaneously, a feat impossible with the sequential nature of IMP.
Discussion and Future Work
A key insight of this work is that determinism is critical; the noise in prior stochastic masking methods was likely their primary failure mode. By making the mask deterministic, CS allows the loss function to decide which weights are necessary based on accuracy and parameter penalties, rather than assuming "small weights are useless."
While CS offers superior performance, it introduces a slight computational overhead (roughly 10% more time per epoch) and requires hyperparameter tuning (like the $\ell_1$ penalty $\lambda$) to hit specific target sparsities. Future work includes extending CS to other domains, such as Natural Language Processing with Transformers, to make the search for winning tickets practical for mobile and transfer learning applications.
Summary updated