Iterative Magnitude Pruning for Sparse Neural Networks

472 concepts353 papers5 proposals

Deep neural networks are notoriously over-parameterized. They contain millions or billions of parameters, many of which can be eliminated after training without hurting performance. However, if you try to train a sparse network with that same compact architecture from scratch, it almost always performs worse than its dense counterpart.

This creates a fundamental paradox: if a sparse configuration is capable of representing the solution, why can't we train it directly?

The answer lies in the optimization landscape. Over-parameterization makes training easier by providing an abundance of random pathways for gradients to flow. To find a sparse network that actually trains well, we need a method to discover these hidden, highly performant pathways within a dense model. The most successful mechanism for doing this is Iterative Magnitude Pruning (IMP).


The Core Intuition: The Lottery Ticket Hypothesis

To understand why IMP works, we must look at the Lottery Ticket Hypothesis, proposed by Jonathan Frankle and Michael Carbin in The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.

The authors suggested that a large, randomly initialized dense network contains a subnetwork initialized in a way that allows it to train exceptionally efficiently. They called these "winning tickets." When you train a dense network, you are essentially running a lottery where only a small subset of parameters are positioned perfectly to solve the task.

If you randomly initialize a sparse network, you are buying only a few lottery tickets, and your chances of winning are low. IMP is a strategy designed to look back at the dense network after it has been trained, identify which parameters won the lottery, and isolate them.


The IMP Protocol: Train, Prune, Rewind

IMP discovers these winning tickets through a cyclical three-step process:

[1. Train Dense Model] ──> [2. Prune Lowest Magnitudes] ──> [3. Rewind to Early State]
         ▲                                                               │
         └───────────────────────────────────────────────────────────────┘
  1. Train: Train the network until it converges.
  2. Prune: Remove a fixed percentage (e.g., 20%) of the weights across the network. The selection criterion is simple: absolute magnitude. Weights closest to zero are discarded under the assumption that they contribute the least to the network's final outputs.
  3. Rewind: Reset the remaining weights back to their values from an earlier point in training (often the original random initialization at epoch 0).

This cycle repeats until the desired level of sparsity is reached.

Why is it iterative?

Why not just prune all the unnecessary weights at once in a single "one-shot" step?

When you prune a weight, you introduce a small error into the network's internal representations. If you prune 80% of the weights simultaneously, these errors compound catastrophically, destroying the model's performance beyond recovery. By pruning gradually (e.g., 20% at a time) and retraining, the remaining weights are given the opportunity to adapt and compensate for the lost connections.


Why Rewinding Works: The Role of Initialization

The most counterintuitive step of IMP is rewinding. If we have already trained a model successfully, why would we throw away that progress and reset the weights to an earlier state?

If you keep the final trained weights after pruning and continue training, the network often gets trapped in a local minimum. It has already converged based on the presence of the weights you just deleted.

Conversely, if you reinitialize the sparse subnetwork with completely new random weights, it fails to train effectively because it loses the "lucky" alignment it had in the dense model.

Rewinding solves this by preserving the exact initial conditions that made those specific parameters successful. In Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask, Hattie Zhou et al. demonstrated that the secret to a winning ticket's success isn't just the exact numerical values of the weights, but their signs. Rewinding preserves these critical directional biases, allowing the subnetwork to follow the same favorable optimization trajectory it discovered during the dense training phase.

Furthermore, these winning tickets are not just overfitted to a single dataset. Ari S. Morcos demonstrated in One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers that winning tickets found on one dataset contain generic inductive biases that allow them to generalize effectively across entirely different datasets and optimizers.


Scaling to Large Models: Learning Rate and Weight Rewinding

While resetting weights to epoch 0 works well for small networks, it breaks down when scaling to massive architectures like Transformers or deep convolutional networks. In these larger spaces, the early optimization path is highly stochastic; resetting all the way to the beginning causes the model to lose its stability.

To fix this, researchers use Weight Rewinding and Learning Rate Rewinding. Instead of resetting parameters to epoch 0, the weights and learning rates are rewound to an early epoch (e.g., epoch 5 or 10) where the network has stabilized but has not yet fully converged.

For instance, Tianlong Chen et al. successfully applied this stabilization technique to large language models in The Lottery Ticket Hypothesis for Pre-trained BERT Networks, discovering universal winning tickets at the pre-training stage that transfer seamlessly to downstream tasks. Similarly, Adriana Fernandez-Lopez et al. utilized learning rate rewinding in SparseVSR: Lightweight and Noise Robust Visual Speech Recognition to strip away 70% of a visual speech model's parameters while maintaining robustness against visual noise.


The Side Effects of Sparsity

While IMP is primarily used to reduce computational overhead and memory footprints, pruning fundamentally alters how a network processes information. This introduces several notable side effects:

By understanding IMP not just as a compression trick, but as a mechanism for isolating the core pathways of a network, researchers can design architectures that are both highly efficient and fundamentally stable.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 374 papers
  2. Version 1 · 2026-10-04 · 554 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_magnitude_pruning_532de21c,
  title = {iterative magnitude pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-magnitude-pruning/rdr_532de21c06054de285499358868cf11c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_iterative_magnitude_pruning_532de21c_v2,
  title = {iterative magnitude pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-magnitude-pruning/rdr_532de21c06054de285499358868cf11c/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake