Deadwooding: Robust Global Pruning for Deep Neural Networks

Sawinder Kaur, Ferdinando Fioretto, Asif SalekinarXiv 2022 · arXiv

Introduction

Deep Neural Networks (DNNs) are typically over-parameterized, containing far more weights than necessary for their primary tasks. Pruning is a standard technique used to remove this "dead wood," enabling models to run on resource-constrained devices. However, traditional pruning often introduces a significant vulnerability to adversarial attacks—small, intentional input perturbations that cause model failure.

Existing robust pruning methods frequently rely on layer-wise strategies, where a fixed percentage of weights is removed from every layer. This approach is often suboptimal, as it ignores the varying importance of different layers. While global pruning offers more flexibility, it can lead to "disconnected" networks where signal flow is interrupted. This paper introduces Deadwooding, a novel global pruning framework that leverages Lagrangian duality to achieve extreme sparsity (up to 99%) without compromising adversarial robustness or network connectivity.

Proposed Method: Deadwooding

Deadwooding treats robust pruning as a constrained optimization problem. By employing a Lagrangian Dual approach, the framework minimizes empirical risk while satisfying specific pruning targets and adversarial tolerance levels.

Key technical components include:

  • Global Pruning with Connectivity Constraints: To prevent network disconnection at high sparsity, Deadwooding incorporates a dedicated loss term during the pruning phase. This ensures an active path remains from input to output even at 99% sparsity.
  • Adversarial Robustness Proxy: Instead of training against specific attacks, the method uses a logit-based proxy to maximize the distance between input samples and decision boundaries, making the model inherently more difficult to perturb.
  • Efficient Fine-Tuning: The framework utilizes "FGSM-looping," a computationally efficient variation of the Fast Gradient Sign Method, and Adversarial Knowledge Distillation to recover accuracy and robustness in the pruned student model.

Experimental Evaluation

The researchers evaluated Deadwooding using the CIFAR-10 dataset on a VGG-16 architecture, comparing it against Hydra, the current state-of-the-art (SoA) for robust pruning.

Layer-Wise vs. Global Flexibility

A key advantage of Deadwooding is its ability to distribute sparsity non-uniformly across the network. As shown in Figure 1, while layer-wise methods like Hydra apply uniform pruning, Deadwooding preserves more weights in critical initial feature extraction and final decision layers, while pruning middle layers more aggressively (up to 99.9%).

Figure 1: Comparison of per-layer pruning percentages for Hydra, Global Pruning, and Deadwooding on VGG-16.

Results and Analysis

Deadwooding consistently outperforms the Hydra baseline across all pruning levels, with the most significant gains observed at extreme sparsity.

Table 1: Performance comparison between Hydra and Deadwooding on CIFAR-10/VGG-16.

At 99% pruning, Deadwooding achieved a benign accuracy (eba) of 80.87% and a robust accuracy (era) of 53.1%. This represents a +7.67% improvement in benign accuracy and a substantial +11.4% increase in robustness compared to Hydra. Ablation studies confirmed that the inclusion of the classification loss during pruning is vital; without it, the network becomes disconnected, and accuracy drops to near-random levels.

Conclusion

Deadwooding demonstrates that global pruning, when combined with connectivity constraints and decision-boundary optimization, can produce highly sparse models that remain robust to adversarial threats. While currently validated on CIFAR-10 and VGG-16 within $L_\infty$ perturbation bounds, future work will explore its scalability to larger datasets like ImageNet and more modern architectures like Vision Transformers, as well as its performance against a broader range of adversarial attack types.

Summary updated

BibTeX
@misc{kaur2022deadwooding,
  author = {Sawinder Kaur and Ferdinando Fioretto and Asif Salekin},
  title = {{Deadwooding: Robust Global Pruning for Deep Neural Networks}},
  year = {2022},
  eprint = {2202.05226},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.05226},
}
Figure