Deep neural networks are notoriously over-parameterized. They contain millions or billions of parameters, many of which can be eliminated after training without hurting performance. However, if you try to train a sparse network with that same compact architecture from scratch, it almost always performs worse than its dense counterpart.
This creates a fundamental paradox: if a sparse configuration is capable of representing the solution, why can't we train it directly?
The answer lies in the optimization landscape. Over-parameterization makes training easier by providing an abundance of random pathways for gradients to flow. To find a sparse network that actually trains well, we need a method to discover these hidden, highly performant pathways within a dense model. The most successful mechanism for doing this is Iterative Magnitude Pruning (IMP).
The Core Intuition: The Lottery Ticket Hypothesis
To understand why IMP works, we must look at the Lottery Ticket Hypothesis, proposed by Jonathan Frankle and Michael Carbin in The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.
The authors suggested that a large, randomly initialized dense network contains a subnetwork initialized in a way that allows it to train exceptionally efficiently. They called these "winning tickets." When you train a dense network, you are essentially running a lottery where only a small subset of parameters are positioned perfectly to solve the task.
If you randomly initialize a sparse network, you are buying only a few lottery tickets, and your chances of winning are low. IMP is a strategy designed to look back at the dense network after it has been trained, identify which parameters won the lottery, and isolate them.
The IMP Protocol: Train, Prune, Rewind
IMP discovers these winning tickets through a cyclical three-step process:
[1. Train Dense Model] ──> [2. Prune Lowest Magnitudes] ──> [3. Rewind to Early State]
▲ │
└───────────────────────────────────────────────────────────────┘
- Train: Train the network until it converges.
- Prune: Remove a fixed percentage (e.g., 20%) of the weights across the network. The selection criterion is simple: absolute magnitude. Weights closest to zero are discarded under the assumption that they contribute the least to the network's final outputs.
- Rewind: Reset the remaining weights back to their values from an earlier point in training (often the original random initialization at epoch 0).
This cycle repeats until the desired level of sparsity is reached.
Why is it iterative?
Why not just prune all the unnecessary weights at once in a single "one-shot" step?
When you prune a weight, you introduce a small error into the network's internal representations. If you prune 80% of the weights simultaneously, these errors compound catastrophically, destroying the model's performance beyond recovery. By pruning gradually (e.g., 20% at a time) and retraining, the remaining weights are given the opportunity to adapt and compensate for the lost connections.
Why Rewinding Works: The Role of Initialization
The most counterintuitive step of IMP is rewinding. If we have already trained a model successfully, why would we throw away that progress and reset the weights to an earlier state?
If you keep the final trained weights after pruning and continue training, the network often gets trapped in a local minimum. It has already converged based on the presence of the weights you just deleted.
Conversely, if you reinitialize the sparse subnetwork with completely new random weights, it fails to train effectively because it loses the "lucky" alignment it had in the dense model.
Rewinding solves this by preserving the exact initial conditions that made those specific parameters successful. In Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask, Hattie Zhou et al. demonstrated that the secret to a winning ticket's success isn't just the exact numerical values of the weights, but their signs. Rewinding preserves these critical directional biases, allowing the subnetwork to follow the same favorable optimization trajectory it discovered during the dense training phase.
Furthermore, these winning tickets are not just overfitted to a single dataset. Ari S. Morcos demonstrated in One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers that winning tickets found on one dataset contain generic inductive biases that allow them to generalize effectively across entirely different datasets and optimizers.
Scaling to Large Models: Learning Rate and Weight Rewinding
While resetting weights to epoch 0 works well for small networks, it breaks down when scaling to massive architectures like Transformers or deep convolutional networks. In these larger spaces, the early optimization path is highly stochastic; resetting all the way to the beginning causes the model to lose its stability.
To fix this, researchers use Weight Rewinding and Learning Rate Rewinding. Instead of resetting parameters to epoch 0, the weights and learning rates are rewound to an early epoch (e.g., epoch 5 or 10) where the network has stabilized but has not yet fully converged.
For instance, Tianlong Chen et al. successfully applied this stabilization technique to large language models in The Lottery Ticket Hypothesis for Pre-trained BERT Networks, discovering universal winning tickets at the pre-training stage that transfer seamlessly to downstream tasks. Similarly, Adriana Fernandez-Lopez et al. utilized learning rate rewinding in SparseVSR: Lightweight and Noise Robust Visual Speech Recognition to strip away 70% of a visual speech model's parameters while maintaining robustness against visual noise.
The Side Effects of Sparsity
While IMP is primarily used to reduce computational overhead and memory footprints, pruning fundamentally alters how a network processes information. This introduces several notable side effects:
- Improved Calibration: In Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Study, Pallavi Mitra et al. found that unstructured weight pruning acts as a regularizer. By removing low-magnitude weights, it mitigates overconfidence, resulting in better uncertainty calibration and stronger robustness to natural corruptions like fog or sensor noise.
- Amplified Bias: Shrinking a model can introduce ethical trade-offs. Hatice Gunes and Samuil Stoychev demonstrated in The Effect of Model Compression on Fairness in Facial Expression Recognition that magnitude-based pruning can disproportionately widen accuracy gaps between demographic groups. When a network loses capacity, it tends to preserve performance on majority data patterns while "forgetting" underrepresented or low-resource cohorts.
By understanding IMP not just as a compression trick, but as a mechanism for isolating the core pathways of a network, researchers can design architectures that are both highly efficient and fundamentally stable.