Preserving Gradient Flow for Pruning at Initialization

This direction is merged into another direction: iterative pruning overhead, gradient flow preservation. Its last version stays readable and citable here.

Version 1 of 123 papers37 conceptsfrozen 2026-10-04

Deep neural networks are typically over-parameterized, containing far more weights than necessary for their final performance. The Lottery Ticket Hypothesis famously suggested that within these dense networks exist sparse "winning tickets"—sub-networks that, if trained in isolation from the start, could match the performance of the full model.

However, finding these tickets traditionally required training the full model to completion, pruning it, and then "rewinding" to the initial weights. This is computationally expensive. A more efficient frontier of research, often called Pruning at Initialization (PaI) or Foresight Pruning, aims to identify these winning tickets before any training occurs. The core insight shared across this research is that a sub-network’s potential is determined not by its initial weight magnitudes, but by its ability to preserve gradient flow.

The Failure of Magnitude at Initialization

In standard post-training pruning, we remove weights with small magnitudes because they contribute little to the final output. But at initialization ($t=0$), weights are typically sampled from a random distribution (like Xavier or Kaiming initialization). In this state, a weight's magnitude is essentially stochastic noise; it carries no information about the weight's eventual importance.

If we prune randomly or by magnitude at the start, we risk creating "dead" neurons or bottlenecks where the gradient signal cannot propagate from the output back to the input layers. As Chaoqi Wang, Guodong Zhang, and Roger B. Grosse argue in Picking Winning Tickets Before Training by Preserving Gradient Flow, the goal of pruning at initialization should be to maintain the "optimization landscape." They introduced GraSP (Gradient Signal Preservation), which identifies weights that, when kept, maximize the flow of the gradient signal through the network.

Aligning Training Dynamics via the NTK

Why does preserving gradient flow work? Recent theoretical work links this intuition to the Neural Tangent Kernel (NTK). The NTK describes how a neural network's outputs change in response to gradient descent updates. If a sparse sub-network has an NTK similar to the original dense model, it will follow a similar training trajectory and reach a similar solution.

Daniel Zügner and colleagues demonstrated in Winning the Lottery Ahead of Time: Efficient Early Network Pruning that by preserving the NTK, a sparse model inherits the convergence properties and generalization behavior of the over-parameterized architecture. This was further refined by Yite Wang and Ruoyu Sun in NTK-SAP, which aligns the eigenspectrum of the sparse NTK with the dense one. By ensuring the "shape" of the learning dynamics is preserved, they achieved high accuracy even at extreme sparsity levels (98%+).

Beyond Weights: Layers and Paths

The principle of gradient flow applies beyond individual connections. Researchers have found that entire structural components can be redundant if their removal doesn't choke the signal:

Why It Works: The Intuition of Trainability

The fundamental reason these methods succeed is that they prioritize trainability over immediate performance.

At the start of training, the loss is high and the model is inaccurate. A magnitude-based pruner sees a "useless" weight and cuts it. However, a gradient-flow pruner—like those used in FedPaI by Zhengwu Liu—asks: "If I change this weight, how much does it help the rest of the network learn?"

By keeping the weights that contribute most to the gradient signal, we ensure the sparse network remains an efficient "information highway." This allows the model to escape the initial random state and converge toward a global minimum as effectively as a dense model, but with a fraction of the computational cost. This shift from "what the weight is" to "what the weight allows the network to do" is the defining characteristic of modern foresight pruning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_preserving_gradient_flow_for_pruning_at__b6c4f819,
  title = {Preserving Gradient Flow for Pruning at Initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-gradient-flow-for-pruning-at-initialization/rdr_b6c4f819bd154248a92db48d5c987fa1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_preserving_gradient_flow_for_pruning_at__b6c4f819_v1,
  title = {Preserving Gradient Flow for Pruning at Initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-gradient-flow-for-pruning-at-initialization/rdr_b6c4f819bd154248a92db48d5c987fa1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}