Preserving Gradient Flow and Neural Tangent Kernels in Network Pruning

22 concepts10 papers1 proposal

Traditional neural network pruning relies on a simple post-training recipe: train a dense network to convergence, identify unimportant weights (usually those with the smallest magnitudes), snip them away, and fine-tune the remaining sparse structure. While effective, this approach requires paying the full computational price of dense training before any sparsity benefits are realized.

To bypass this overhead, researchers have increasingly focused on "pruning at initialization" or early in the training cycle. However, applying traditional heuristics early in training fails catastrophically. At initialization, weight magnitudes are purely random, and the initial loss is merely stochastic noise.

A powerful alternative paradigm has emerged to solve this problem: pruning by preserving optimization dynamics. Instead of evaluating what a network has already learned, these methods treat the network as a dynamical system and prune parameters to preserve its capacity to learn. By maintaining healthy gradient flow and aligning the network's underlying Neural Tangent Kernel (NTK), a sparse subnetwork can mimic the exact training trajectory and convergence behavior of its dense counterpart from day one.


The Pitfalls of Loss and Magnitude Heuristics

Why can't we simply prune weights that have the least impact on the immediate loss? As Chaoqi Wang et al. observe in Picking Winning Tickets Before Training by Preserving Gradient Flow, the initial loss value of a randomly initialized network contains no meaningful signal about which pathways will become important.

Furthermore, attempting to aggressively maximize learning acceleration during early pruning can backfire. In A Gradient Flow Framework For Analyzing Network Pruning, Ekdeep Singh Lubana et al. demonstrate a fundamental conflict between gradient acceleration and loss preservation. When a pruning algorithm tries to maximize the gradient norm to force faster learning, it frequently strips away filters and parameters in the early layers that are vital for long-term loss reduction. This causes an irreversible collapse in the network's learning capacity.

The solution is not to maximize or alter the gradient, but to preserve its existing flow, ensuring stable second-order evolution dynamics throughout the network.


Preserving the Flow of Signals

To keep a pruned network trainable, the optimization signal must propagate backward through the layers without exploding or vanishing.

In their seminal work on Gradient Signal Preservation (GraSP), Chaoqi Wang et al. shifted the pruning objective from minimizing loss change to minimizing the change in the gradient norm. This ensures that the gradient signal remains robust after sparsification. This principle scales remarkably well across different structural granularities and architectures:

  • Layer Pruning: When removing entire redundant layers at initialization, Artur Jordão et al. demonstrate in When Layers Play the Lottery, all Tickets Win at Initialization that metrics prioritizing signal flow—specifically those tracking the Hessian-gradient product—are vastly superior to weight magnitudes. They keep the global gradient path intact even at extreme compression levels.
  • Generative Models: In the context of diffusion models, Ben Wan et al. show in Pruning for Sparse Diffusion Models based on Gradient Flow that a progressive soft pruning strategy guided by gradient flow prevents the "quality crash" typical of one-shot pruning, allowing the model to recover and converge rapidly during fine-tuning.
  • Local Optimization Alignment: Even when upgrading magnitude-based approaches, incorporating the structural context of surrounding layers helps. Sejun Park et al. show in Lookahead: A Far-sighted Alternative of Magnitude-based Pruning that factoring in the norms of preceding and succeeding connections respects the natural "balancedness" property of stochastic gradient descent, preserving structural integrity.

The Theoretical Foundation: Neural Tangent Kernel Alignment

The intuition of preserving gradient flow finds its formal mathematical backing in Neural Tangent Kernel (NTK) theory. The NTK describes how a neural network's outputs change in response to gradient updates during training. If a sparse subnetwork has an NTK that matches the dense network, it will learn in the exact same way.

As Daniel Zügner et al. establish in Winning the Lottery Ahead of Time: Efficient Early Network Pruning, explicitly preserving gradient flow during sparsification serves as a direct approximation for preserving the NTK. By maintaining this kernel stability, the sparse model inherits the convergence guarantees and generalization behaviors of the original over-parameterized architecture.

Dense Network ───► [Compute Dense NTK] ───┐
                                          ▼
                                  [Spectrum Alignment] ───► Sparse Subnetwork with
                                          ▲                 Identical Learning Dynamics
Sparse Network ──► [Compute Sparse NTK] ──┘

To achieve extreme sparsities (often exceeding 98%) without losing accuracy, recent methods focus on aligning the eigenspectrum of the NTK. The spectrum dictates how quickly a network learns along different data dimensions.

In NTK-SAP: Improving neural network pruning by aligning training dynamics, Yite Wang et al. introduce a foresight pruning method that explicitly aligns this spectrum, making the process both weight- and data-agnostic. Similarly, Leonardo Iurada et al. propose Path eXclusion (PX) in Finding Lottery Tickets in Vision Models via Data-Driven Spectral Foresight Pruning, which incorporates data-driven sample pathways to align the NTK spectrum, successfully preserving the transferability of large pre-trained models.

Providing a unifying theoretical lens to this mechanism, Hoang Pham et al. introduce the Graphon NTK framework in The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis. They prove that random pruning acts merely as a scalar reduction of the NTK—which preserves the dynamics but slows down training speed. In contrast, optimization-aligned structured pruning actively reshapes the kernel's geometry, prioritizing specific information pathways so the sparse network can train with maximum efficiency.


Extension to Optimization: Gradient Directional Integrity

The core philosophy of this research direction—that preserving the fidelity of gradient dynamics is paramount—extends beyond structural pruning into the optimization process itself.

For instance, during backpropagation, computing exact gradients at every step is highly expensive. In ReSprop: Reuse Sparsified Backpropagation, Negar Goli et al. exploit the fact that a vast majority of gradients change very little between consecutive iterations. Rather than completely pruning small gradients (which introduces optimization instability), they use hybrid updates to reuse past gradients. This maintains the directional integrity of the overall gradient vector.

Just as spectral NTK alignment ensures a structurally sparse network behaves like a dense one, maintaining directional alignment in the gradient space allows approximated backpropagation to achieve massive computational speedups with virtually no loss in final accuracy.

Key Foundational Work

A Gradient Flow Framework For Analyzing Network Pruning (2021) — contributed 4 concept elements to this direction.
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020) — contributed 2 concept elements to this direction.
Lookahead: A Far-sighted Alternative of Magnitude-based Pruning (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_flow_preservation_dc8ee496,
  title = {gradient flow preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-flow-preservation/rdr_dc8ee4961083404385c4d0c44fb24ea3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_flow_preservation_dc8ee496_v1,
  title = {gradient flow preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-flow-preservation/rdr_dc8ee4961083404385c4d0c44fb24ea3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake