To deploy massive neural networks on edge devices, we must compress them. The most common and intuitive approach is Magnitude-Based Pruning (MBP): calculate the absolute value or $L_1$-norm of the weights, and discard the smallest ones. The underlying assumption is simple: if a weight's absolute value is close to zero, its contribution to the network's output must be negligible.
However, this intuitive leap is fundamentally flawed. A parameter's size does not dictate its structural or functional utility. Across diverse domains—from large language models to multi-modal vision encoders—relying strictly on weight magnitude as a proxy for importance introduces severe biases, often discarding critical features while retaining redundant noise.
The Geometry of Curvature vs. Size
The core failure of magnitude-based pruning stems from treating a highly non-linear loss landscape as a simple linear system. In complex networks, small weights can be linchpins of critical computational paths, while large weights can be entirely redundant.
To understand why, we must look at the loss landscape's curvature. If we prune a weight, we are effectively shifting its value to zero. The impact of this shift on the network's loss function ($\mathcal{L}$) can be approximated using a Taylor expansion:
$$\Delta \mathcal{L} \approx g_i \Delta w_i + \frac{1}{2} H_{ii} (\Delta w_i)^2$$
where $g_i$ is the gradient and $H_{ii}$ is the corresponding diagonal element of the Hessian matrix (the second-order derivative).
Magnitude pruning assumes that if $w_i$ is small, $\Delta w_i$ is small, and thus $\Delta \mathcal{L}$ must be small. But this ignores the Hessian curvature $H_{ii}$. A parameter with a tiny absolute value sitting along a highly steep direction of the loss surface (large $H_{ii}$) will cause a catastrophic spike in loss if removed. Conversely, a large weight sitting in a flat valley can be dropped with zero practical impact.
As demonstrated by Xing et al. in Probabilistic Connection Importance Inference and Lossless Compression of Deep Neural Networks, magnitude metrics frequently misidentify functional importance in non-linear systems. To resolve this, second-order frameworks like WoodFisher: Efficient Second-Order Approximation for Neural Network Compression compute efficient approximations of the inverse Hessian matrix to capture how weights interact. By evaluating how weights collectively influence the loss rather than checking their sizes in isolation, these methods avoid removing individually small but collectively vital parameters. Molchanov further formalizes this in Importance Estimation for Neural Network Pruning by utilizing gradient-based Taylor expansions, proving that sensitivity to perturbations is a far more reliable pruning signal than static weight values.
The Blindspot of Lazy Training and Fine-Tuning
The failure of magnitude heuristics becomes acute when compressing large pre-trained models for downstream tasks. When fine-tuning massive models, optimization often occurs within a "lazy training" regime. In this state, the network's weights do not shift significantly from their initial pre-trained values to minimize the downstream loss.
Because the weights remain close to their initialization, their absolute magnitudes reflect the statistical distribution of the pre-training data, not their utility for the downstream task. As Liu et al. demonstrate in Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!, magnitude-based pruning heuristics fail catastrophically when evaluated on complex, non-saturated benchmarks. Because the task-specific signal is buried under the massive, static pre-trained weight values, simple magnitude sorting acts as a noisy, task-agnostic lottery.
This limitation is also highlighted by Chen et al. in DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models. When a model is adapted to a narrow target domain, the parameters critical for that specific task might have received small updates and possess small absolute magnitudes. A task-agnostic magnitude prune will aggressively strip these parameters away, destroying the model's specialized downstream capabilities.
Structural Context and Information Flow
Modern architectures introduce structural dependencies and normalization layers that completely decouple weight magnitude from functional importance.
Consider three distinct architectural scenarios where local magnitude fails:
- Normalized Circuits: In architectures with constrained or normalized parameters—such as sum units in probabilistic circuits—Dang et al. show that local weight values are misleading. Because weights are bounded by normalization, a locally small value can still carry a globally significant portion of the generative information flow (Sparse Probabilistic Circuits via Pruning and Growing).
- Global Attention Mechanisms: In Vision Transformers (ViTs), complex multi-head attention blocks interact globally. Yang et al. demonstrate in Global Vision Transformer Pruning with Hessian-Aware Saliency that pruning based on isolated weight magnitudes disrupts the dynamic functional connectivity across layers. A structural component cannot be judged by its local norm without accounting for the global sensitivity of the loss function to that entire block.
- Cross-Modal Alignment: In multi-modal models like CLIP, text and visual representations must map to a shared embedding space. Lin et al. reveal in MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric that the delicate alignment between text and image encoders often relies on low-magnitude boundary weights. Slicing connections based purely on size breaks this cross-modal coordination, severely degrading multi-modal reasoning while leaving single-modality features intact.
Functional Redundancy and Hidden Vulnerabilities
Perhaps the most counterintuitive limitation of magnitude pruning is that it frequently retains parameters that are completely useless while discarding those responsible for network stability and safety.
In a striking reversal of conventional wisdom, Yu proves in Scaling Up Exact Neural Network Compression by ReLU Stability that connections with exceptionally high-magnitude weights are frequently functionally redundant. This occurs when large weights feed into neurons that are permanently stable (e.g., always inactive or always active across the data distribution). Magnitude pruning blindly preserves these heavy, dead pathways while stripping away lower-magnitude weights that actively drive the network's logic.
Furthermore, magnitude pruning can compromise model robustness. Zheng et al. demonstrate that small weights are often essential for maintaining a model's defense against adversarial attacks (Robust Lottery Tickets for Pre-trained Language Models). Because magnitude heuristics are not guided by an adversarial objective, they systematically discard the subtle, low-magnitude stabilizing paths that prevent input perturbations from causing chaotic output swings.
This blindspot extends to security vulnerabilities. As shown by the authors of RIBAC: Towards Robust and Imperceptible Backdoor Attack against Compact DNN, malicious backdoor pathways injected into a network do not necessarily align with large weight values. Standard magnitude pruning fails to disrupt these hidden vulnerabilities because it evaluates parameters by their scale rather than their semantic contribution to the network's underlying reasoning.