Pruning Neural Networks via Functional Sensitivity

16 concepts10 papers1 proposal

Standard neural network pruning often relies on a simple heuristic: magnitude. If a weight or an activation is small, we assume it does not matter and snip it away. However, magnitude is a poor proxy for importance. A large weight might feed into a dead neuron, while a tiny weight might be the sole anchor for a rare but critical feature.

Instead of asking "how big is this parameter?", a more powerful approach asks: "how much does the network's behavior change if this parameter is perturbed?" This question defines the core mechanism of sensitivity-based pruning. By shifting the focus from static values to functional impact, this paradigm allows us to compress networks drastically while maintaining rigorous performance guarantees.

The Core Intuition: Sensitivity as a Derivative

To understand why sensitivity-based pruning works, imagine a network as a complex mathematical landscape. The sensitivity of a specific component (a weight, a neuron, or an attention head) is a measure of how much the landscape shifts when that component is modified.

As shown by Bragagnolo et al., we can quantify this importance by directly measuring the variation in the network's output relative to changes in a specific neuron's activity. If a neuron can fluctuate wildly without altering the final predictions, it is functionally redundant—regardless of how large its weights are.

This distinction becomes especially vital in knowledge-intensive tasks like factual recall or reasoning. Zhao et al. demonstrated that a neuron's empirical gradient—how the final output probability responds to a perturbation—is a far more reliable indicator of its contribution to a specific skill than its static activation level. A neuron might fire frequently (high activation), but if its gradient is near zero, it acts as background noise rather than a decision driver.

From Sensitivity to Coresets: Provable Pruning

How do we translate these sensitivity scores into a smaller network without breaking its internal representations? The answer lies in the mathematical framework of coresets—small, weighted subsets of a larger structure that provably preserve the properties of the whole.

Instead of deterministically cutting all parameters below a certain sensitivity threshold, the coreset approach treats sensitivity as a probability distribution for importance sampling. The process works as follows:

  1. Compute Sensitivity: Calculate the sensitivity of each parameter or neuron, which bounds its maximum possible contribution to the subsequent layer's output.
  2. Sample Probabilistically: Select a subset of parameters where components with higher sensitivity have a proportionally higher chance of being retained.
  3. Reweight: To prevent the layer's output from shrinking, scale up the weights of the selected parameters inversely to their selection probability.

This reweighting step is the secret to why the method works. If a neuron has a low sensitivity score, it has a low probability of being picked. If it is picked, its weight is scaled up significantly. Mathematically, this ensures that the expected output of the pruned layer remains completely unbiased and identical to the original version.

Using this framework, Mussay et al. (iclr 2020) and Mussay et al. (arxiv 2020) designed data-independent structured pruning algorithms. Because their sensitivity metrics account for the worst-case impact on any potential future input, they provide the first rigorous, provable guarantees on the trade-off between compression and approximation error.

To tighten these bounds for practical deployment, Baykal et al. (iclr 2019) and Baykal et al. (2022) introduced data-dependent sensitivity analysis. By evaluating sensitivity over a representative validation set or a small batch of input data, their methods capture how parameters behave on real data distributions, leading to massive compression ratios with minimal accuracy loss.

Scaling to Transformers via Gradient Approximations

For massive architectures like Large Language Models (LLMs), computing exact combinatorial sensitivity or tracking layer-by-layer bounds is computationally intractable. Modern implementations scale this idea by approximating sensitivity using first-order derivatives (gradients) relative to the final loss function.

During knowledge distillation, for example, maintaining teacher-student alignment is notoriously difficult. Liang et al. solved this by using a first-order Taylor expansion to calculate the sensitivity of the loss function to specific neurons. Pruning neurons based on this loss-centric sensitivity allows the student model to retain the teacher's functional capabilities far better than traditional magnitude-based distillation.

This gradient-informed approach is equally powerful for dynamic, runtime compression. Akhauri et al. utilized gradient-informed sensitivity metrics to determine which activations and attention heads are critical to the final loss for a specific input context. By targeting components that actively shape the output loss rather than those with the largest raw magnitudes, they achieved high contextual sparsity and faster inference without degrading model accuracy.

Why It Works: The Geometry of Overparameterization

The remarkable success of sensitivity-based sampling is deeply tied to the underlying geometry of deep neural networks. Theoretical work by Fang et al. shows that wide, overparameterized networks naturally contain a vast sea of redundant parameters. However, embedded within this landscape is a subset of highly effective neurons that drive the target function representation. Importance sampling acts as a mathematical sieve, isolating these active features from the wasted parameters.

Furthermore, deep models often allocate specific semantic concepts to localized units. As Lin et al. discovered in self-supervised speech Transformers, individual feedforward units act as highly specialized "property neurons" that detect traits like phonemes, gender, or pitch.

Standard magnitude pruning frequently discards these units if their static weights happen to be small, permanently erasing specific downstream capabilities. Sensitivity metrics naturally protect these specialized property detectors because they measure the functional damage that would occur if the neuron were removed. By preserving these critical nodes, sensitivity-informed pruning maintains the rich, multi-faceted intelligence of the original model at a fraction of the computational cost.

Key Foundational Work

Data-Independent Neural Pruning via Coresets (2020) — contributed 2 concept elements to this direction.
Data-Independent Structured Pruning of Neural Networks via Coresets (2020) — contributed 1 concept elements to this direction.
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sensitivity_based_pruning_8ecc7585,
  title = {sensitivity-based pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sensitivity-based-pruning/rdr_8ecc758506df4f028aa1ce289037b4c2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sensitivity_based_pruning_8ecc7585_v1,
  title = {sensitivity-based pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sensitivity-based-pruning/rdr_8ecc758506df4f028aa1ce289037b4c2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake