Elimination-compensation pruning for fully-connected neural networks

Enrico Ballini, Luca Muscarnera, Alessio Fumagalli, Anna Scotti, Francesco RegazzoniarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces a novel neural network pruning technique that compensates for removed weights by adjusting adjacent biases. This "elimination-compensation" approach aims to preserve network accuracy more effectively than traditional methods by considering the interplay between weights and biases, leading to significant compression with minimal performance degradation.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are incredibly powerful but often come with a massive number of parameters. This leads to challenges in training efficiency, memory storage, and inference speed, especially for deployment on resource-constrained devices. Pruning techniques aim to reduce this parameter count by removing "unimportant" weights, but a key assumption is that these weights have a small impact on the network's output. This paper questions this assumption and proposes a more nuanced approach that considers how removing a weight might be compensated by adjusting other parameters, specifically biases, to maintain performance.

Methods

The core of this paper is the "elimination-compensation" pruning method. Instead of simply zeroing out weights, the authors propose to:

  • Compensate weight removal with bias adjustment: When a weight $W_{ij}^l$ is considered for removal, its impact is mitigated by adjusting the corresponding bias $b_i^l$. This adjustment, denoted as $\Delta b_i^l$, is incorporated into the layer's operation: $z_i^l = \sigma^l(M_{ij}^l \odot W_{ij}^l z_{i-1}^{l-1} + b_i^l + \Delta b_i^l)$.
  • Define a novel importance measure: The importance of a weight $W_{ij}^l$ is quantified by considering the expected squared discrepancy in the network's output when the weight is removed and the bias is optimally adjusted. This is achieved through a Taylor expansion of the network's output with respect to the weight and bias, leading to an analytical expression for the optimal bias adjustment and the weight's importance.
  • Efficient computation: The proposed method leverages automatic differentiation to efficiently compute the necessary derivatives for determining the optimal bias compensation and importance scores. This is crucial because a naive weight-by-weight evaluation would be computationally prohibitive. The authors highlight that their method's computational cost is comparable to a single backpropagation step, making it significantly more affordable than brute-force or some other importance estimation methods.

Datasets

The paper evaluates its proposed method on two main types of datasets:

  • MNIST: A standard benchmark for image classification tasks.
  • Diffusion-Sorption PDE dataset: A dataset derived from solutions of a 1D partial differential equation with added noise, used to test the method's robustness in scientific machine learning applications.

The authors note that for the PDE dataset, they used a smaller subset of the original data to manage the computational cost of the extensive comparisons.

Key Insights and Ideas

  • Interplay of weights and biases is crucial: The paper argues that simply removing weights without considering their interaction with biases is suboptimal. The proposed method explicitly accounts for this interaction.
  • Bias compensation can preserve accuracy: By adjusting biases, the network can better tolerate the removal of weights, leading to higher compression ratios without significant loss in accuracy.
  • Output-based importance is effective: The method defines importance based on the discrepancy in the network's output, rather than solely on the loss function. This is argued to be more robust, especially at convergence where loss gradients can be dominated by numerical noise.
  • Computational efficiency is a major advantage: The proposed method's ability to compute importance scores efficiently, comparable to a single backpropagation pass, makes it practical for large-scale pruning.

Key Results

  • Significant Compression: The proposed "elimination-compensation" pruning method consistently achieves lower test losses compared to other popular pruning strategies (Magnitude, Gradient-Magnitude, Random, Non-linear) across various pruning ratios, even when removing up to 80-90% of the weights.
  • Robustness to Noise: In the Diffusion-Sorption PDE case, the method demonstrated robustness to different levels of added noise, maintaining performance where other methods might struggle.
  • Competitive with Non-linear (but much faster): While the brute-force "Non-linear" method occasionally performed slightly better, it was orders of magnitude more computationally expensive, making the proposed method a far more practical choice.

Limitations and Future Work

  • Focus on Fully-Connected Networks: The current work is limited to real-valued fully-connected neural networks. The authors suggest that extending the method to other architectures (e.g., convolutional or recurrent networks) would be a valuable future direction.
  • Further Exploration of Importance Metrics: While the output-based importance metric is shown to be effective, further research into its theoretical underpinnings and potential variations could be beneficial.
  • Timing Comparisons: The authors mention that direct timing comparisons are heavily influenced by hardware and code optimizations, and thus were not provided. Future work could include more detailed performance profiling.

Summary updated

BibTeX
@misc{ballini2026elimination,
  author = {Enrico Ballini and Luca Muscarnera and Alessio Fumagalli and Anna Scotti and Francesco Regazzoni},
  title = {{Elimination-compensation pruning for fully-connected neural networks}},
  year = {2026},
  eprint = {2602.20467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20467},
}
Figure