Introduction
Deep Neural Networks (DNNs) are typically over-parameterized, containing far more weights than necessary for their primary tasks. Pruning is a standard technique used to remove this "dead wood," enabling models to run on resource-constrained devices. However, traditional pruning often introduces a significant vulnerability to adversarial attacks—small, intentional input perturbations that cause model failure.
Existing robust pruning methods frequently rely on layer-wise strategies, where a fixed percentage of weights is removed from every layer. This approach is often suboptimal, as it ignores the varying importance of different layers. While global pruning offers more flexibility, it can lead to "disconnected" networks where signal flow is interrupted. This paper introduces Deadwooding, a novel global pruning framework that leverages Lagrangian duality to achieve extreme sparsity (up to 99%) without compromising adversarial robustness or network connectivity.
Proposed Method: Deadwooding
Deadwooding treats robust pruning as a constrained optimization problem. By employing a Lagrangian Dual approach, the framework minimizes empirical risk while satisfying specific pruning targets and adversarial tolerance levels.
Key technical components include:
- Global Pruning with Connectivity Constraints: To prevent network disconnection at high sparsity, Deadwooding incorporates a dedicated loss term during the pruning phase. This ensures an active path remains from input to output even at 99% sparsity.
- Adversarial Robustness Proxy: Instead of training against specific attacks, the method uses a logit-based proxy to maximize the distance between input samples and decision boundaries, making the model inherently more difficult to perturb.
- Efficient Fine-Tuning: The framework utilizes "FGSM-looping," a computationally efficient variation of the Fast Gradient Sign Method, and Adversarial Knowledge Distillation to recover accuracy and robustness in the pruned student model.
Experimental Evaluation
The researchers evaluated Deadwooding using the CIFAR-10 dataset on a VGG-16 architecture, comparing it against Hydra, the current state-of-the-art (SoA) for robust pruning.
Layer-Wise vs. Global Flexibility
A key advantage of Deadwooding is its ability to distribute sparsity non-uniformly across the network. As shown in Figure 1, while layer-wise methods like Hydra apply uniform pruning, Deadwooding preserves more weights in critical initial feature extraction and final decision layers, while pruning middle layers more aggressively (up to 99.9%).
Results and Analysis
Deadwooding consistently outperforms the Hydra baseline across all pruning levels, with the most significant gains observed at extreme sparsity.
At 99% pruning, Deadwooding achieved a benign accuracy (eba) of 80.87% and a robust accuracy (era) of 53.1%. This represents a +7.67% improvement in benign accuracy and a substantial +11.4% increase in robustness compared to Hydra. Ablation studies confirmed that the inclusion of the classification loss during pruning is vital; without it, the network becomes disconnected, and accuracy drops to near-random levels.
Conclusion
Deadwooding demonstrates that global pruning, when combined with connectivity constraints and decision-boundary optimization, can produce highly sparse models that remain robust to adversarial threats. While currently validated on CIFAR-10 and VGG-16 within $L_\infty$ perturbation bounds, future work will explore its scalability to larger datasets like ImageNet and more modern architectures like Vision Transformers, as well as its performance against a broader range of adversarial attack types.
Summary updated