Deep neural networks are typically over-parameterized, containing far more weights than necessary for their final performance. The Lottery Ticket Hypothesis famously suggested that within these dense networks exist sparse "winning tickets"—sub-networks that, if trained in isolation from the start, could match the performance of the full model.
However, finding these tickets traditionally required training the full model to completion, pruning it, and then "rewinding" to the initial weights. This is computationally expensive. A more efficient frontier of research, often called Pruning at Initialization (PaI) or Foresight Pruning, aims to identify these winning tickets before any training occurs. The core insight shared across this research is that a sub-network’s potential is determined not by its initial weight magnitudes, but by its ability to preserve gradient flow.
The Failure of Magnitude at Initialization
In standard post-training pruning, we remove weights with small magnitudes because they contribute little to the final output. But at initialization ($t=0$), weights are typically sampled from a random distribution (like Xavier or Kaiming initialization). In this state, a weight's magnitude is essentially stochastic noise; it carries no information about the weight's eventual importance.
If we prune randomly or by magnitude at the start, we risk creating "dead" neurons or bottlenecks where the gradient signal cannot propagate from the output back to the input layers. As Chaoqi Wang, Guodong Zhang, and Roger B. Grosse argue in Picking Winning Tickets Before Training by Preserving Gradient Flow, the goal of pruning at initialization should be to maintain the "optimization landscape." They introduced GraSP (Gradient Signal Preservation), which identifies weights that, when kept, maximize the flow of the gradient signal through the network.
Aligning Training Dynamics via the NTK
Why does preserving gradient flow work? Recent theoretical work links this intuition to the Neural Tangent Kernel (NTK). The NTK describes how a neural network's outputs change in response to gradient descent updates. If a sparse sub-network has an NTK similar to the original dense model, it will follow a similar training trajectory and reach a similar solution.
Daniel Zügner and colleagues demonstrated in Winning the Lottery Ahead of Time: Efficient Early Network Pruning that by preserving the NTK, a sparse model inherits the convergence properties and generalization behavior of the over-parameterized architecture. This was further refined by Yite Wang and Ruoyu Sun in NTK-SAP, which aligns the eigenspectrum of the sparse NTK with the dense one. By ensuring the "shape" of the learning dynamics is preserved, they achieved high accuracy even at extreme sparsity levels (98%+).
Beyond Weights: Layers and Paths
The principle of gradient flow applies beyond individual connections. Researchers have found that entire structural components can be redundant if their removal doesn't choke the signal:
- Layer Pruning: Artur Jordão and his team showed in When Layers Play the Lottery, all Tickets Win at Initialization that metrics prioritizing signal flow can identify entire redundant layers before training, significantly reducing carbon emissions and training time.
- Data-Driven Paths: Leonardo Iurada and Marco Ciccone introduced Path eXclusion (PX) in Finding Lottery Tickets in Vision Models via Data-Driven Spectral Foresight Pruning. PX accounts for how specific data samples flow through the network, ensuring that the paths necessary for processing the dataset are preserved.
Why It Works: The Intuition of Trainability
The fundamental reason these methods succeed is that they prioritize trainability over immediate performance.
At the start of training, the loss is high and the model is inaccurate. A magnitude-based pruner sees a "useless" weight and cuts it. However, a gradient-flow pruner—like those used in FedPaI by Zhengwu Liu—asks: "If I change this weight, how much does it help the rest of the network learn?"
By keeping the weights that contribute most to the gradient signal, we ensure the sparse network remains an efficient "information highway." This allows the model to escape the initial random state and converge toward a global minimum as effectively as a dense model, but with a fraction of the computational cost. This shift from "what the weight is" to "what the weight allows the network to do" is the defining characteristic of modern foresight pruning.