Curvature Informed Pruning for Deep Neural Networks

46 concepts36 papers1 proposal

The standard approach to neural network pruning is magnitude-based: simply remove the weights with the smallest absolute values. While intuitive, this heuristic assumes that a weight's size is a perfect proxy for its importance. However, as models scale and tasks become more complex, this assumption breaks down. Liu et al. demonstrate that magnitude pruning fails catastrophically on difficult reasoning tasks, even at low sparsity levels, because it ignores the underlying geometry of the loss landscape.

Curvature-informed pruning addresses this by using the Hessian matrix—the second-order derivative of the loss function—to identify which parameters can be removed with the least impact on model performance.

The Geometry of Importance

To understand why curvature matters, consider the Taylor expansion of the loss function $L$ around the current weights $w$. If we remove a weight (setting it to zero), we are introducing a perturbation $\delta w$. The resulting change in loss $\Delta L$ can be approximated as:

$$\Delta L \approx g^T \delta w + \frac{1}{2} \delta w^T H \delta w$$

At a local minimum, the gradient $g$ is zero, leaving the second-order term—the Hessian $H$—as the primary determinant of sensitivity. As Paul et al. explain, pruning weights along "flat" directions (low curvature) preserves performance, while pruning along "sharp" directions (high curvature) leads to significant error spikes.

This geometric perspective explains why small weights aren't always redundant. Dong et al. argue that small weights can be structurally vital if they reside in high-curvature regions of the loss surface. Conversely, large weights in flat regions may be safer to remove than small weights in sharp regions. By calculating a "saliency" or sensitivity score based on this curvature, researchers can identify the true "winning tickets" within a network (Zhang et al.).

Scaling Second-Order Information

The challenge with Hessian-based pruning is computational. For a model with $N$ parameters, the Hessian contains $N^2$ elements, making it impossible to compute or store for modern Large Language Models (LLMs). To make this tractable, the field relies on several key approximations:

  1. Diagonal and Block-Diagonal Approximations: Many methods, such as those discussed by Iurada et al. and Xie et al., use only the diagonal elements of the Hessian. This treats each weight as independent, reducing the complexity to $O(N)$.
  2. The Empirical Fisher Information Matrix: Alistarh and Lee et al. use the Empirical Fisher matrix as a proxy for the Hessian. This captures the correlation structure between weights without requiring the full second-derivative calculation.
  3. The Woodbury Identity: The WoodFisher method (Alistarh) utilizes the Woodbury matrix identity to efficiently estimate the inverse Hessian, which is required to update the remaining weights to compensate for the pruned ones.

These mathematical "shortcuts" allow second-order methods to scale. For instance, SparseGPT (Frantar et al.) can prune models with over 175 billion parameters in a single "one-shot" pass by solving a local reconstruction problem informed by the Hessian.

Applications Across Architectures

Curvature-informed pruning has proven effective across diverse domains:

  • Large Language Models: Beyond SparseGPT, the Optimal BERT Surgeon (Kurtic et al.) uses Hessian information to achieve massive speedups on CPUs by combining pruning with layer dropping and quantization.
  • Vision Transformers (ViTs): Yang et al. and Zheng et al. use Hessian-based saliency to prune attention heads and MLP layers globally, accounting for the complex dependencies between different transformer components.
  • Federated Learning: In decentralized settings, Zhang et al. and Liu et al. use the Hessian to determine layer-specific pruning rates, allowing resource-constrained edge devices to participate in training without sacrificing global model accuracy.
  • Specialized Hardware: Even in hardware acceleration, Yan et al. use second-derivative calculations to identify which weights in computing-in-memory accelerators require the highest precision, optimizing the "write-verify" process.

Why It Works: The Intuition

The fundamental reason curvature-informed pruning outperforms magnitude-based methods is its objective-awareness. Magnitude pruning is a "blind" operation—it looks only at the weights themselves. Curvature-informed pruning looks at the loss function.

By analyzing the Hessian, the algorithm asks: "If I remove this weight, how much will the model's error actually change?" This allows the model to preserve small but critical connections that maintain the network's internal representations (Dong et al.). As Alistarh et al. demonstrate, these heuristics are formally equivalent to iterative optimization steps, providing a theoretical bridge between simple pruning and rigorous sparse recovery.

Key Foundational Work

Node Splitting: A Constructive Algorithm for Feed-Forward Neural Networks (1991) — contributed 1 concept elements to this direction.
Second Order Derivatives for Network Pruning: Optimal Brain Surgeon (1992) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_curvature_informed_pruning_14631413,
  title = {curvature-informed pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curvature-informed-pruning/rdr_14631413b73c45639627910b2e9d234e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_curvature_informed_pruning_14631413_v1,
  title = {curvature-informed pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curvature-informed-pruning/rdr_14631413b73c45639627910b2e9d234e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake