Abstract
Deep Neural Network (DNN) pruning is a critical technique for deploying complex models on resource-constrained edge devices. However, existing adaptive pruning methods often rely on computationally expensive external monitors or reinforcement learning agents. This paper introduces Protective Self-Adaptive Pruning (PSAP), a "monitor-less" approach that leverages a model's internal weight sparsity and gradients to guide the compression process. PSAP incorporates a self-adaptive mechanism for dynamic layer-wise pruning and a protective reconstruction mechanism to recover accidentally pruned critical filters, achieving state-of-the-art compression ratios with minimal accuracy loss.
Introduction
As Deep Neural Networks grow in complexity, the need for efficient compression techniques like pruning becomes paramount for mobile and edge deployment. While early pruning methods utilized fixed ratios across all layers, modern adaptive methods recognize that different layers possess varying degrees of redundancy.
Despite their effectiveness, current adaptive methods typically depend on external monitors—such as Reinforcement Learning (RL) agents or differentiable masks—to determine layer-specific pruning schedules. These monitors are often "black boxes" that add significant training complexity and computational overhead. Furthermore, aggressive pruning can lead to "pulse gradients"—massive spikes in gradient values caused by Batch Normalization (BN) when critical information is zeroed out—resulting in unrecoverable accuracy drops. PSAP addresses these challenges by enabling the model to "self-regulate" its own compression.
The PSAP Framework
PSAP operates within an Iterative Prune-Train (IPT) framework but replaces external decision-makers with two internal mechanisms that utilize the model's own state.
Self-Adaptive Mechanism
Instead of relying on an external agent, PSAP uses the Weight Sparsity Ratio (WSR) of each layer as an innate indicator of its importance. The underlying logic is that layers which naturally become sparser during the training process are likely more redundant. PSAP dynamically adjusts the pruning ratio for each layer based on its current WSR before every pruning step, allowing the architecture to evolve organically toward a sparse state.
Protective Reconstruction Mechanism
To mitigate the destructive nature of pruning, PSAP introduces a safety net called Protective Reconstruction. After a pruning step, the algorithm monitors the gradients for anomalies. If a "pulse gradient" (an abnormally high spike) is detected in association with a pruned filter, the system identifies that filter as critical and reloads its previous weights. This prevents the mathematical singularities in BN layers that occur when essential feature maps are removed, ensuring training stability.
Experimental Results
The researchers evaluated PSAP across several standard benchmarks, demonstrating its robustness even on architectures that are notoriously difficult to compress.
- CIFAR-10 (ResNet-56): PSAP achieved 93.74% accuracy (a 0.15% increase over the baseline) while reducing FLOPs by nearly 50%. Even at extreme compression levels (~80% FLOPs reduction), it maintained a high accuracy of 90.76%.
- ImageNet (ResNet-50): The method achieved 76.83% accuracy (0.18% higher than the unpruned baseline) with a 36.6% reduction in FLOPs, outperforming popular methods like DCP and PFS.
- ImageNet (MobileNetV2): On this already-compact architecture, PSAP achieved 71.4% accuracy with a 31% FLOPs reduction, surpassing both MetaPruning and AMC in efficiency and performance.
Discussion and Conclusion
PSAP demonstrates that a model’s internal state—specifically its weight sparsity and gradient behavior—is a sufficient proxy for layer importance, rendering expensive external search algorithms unnecessary. By turning pruning into a reversible process during the search phase, PSAP ensures that the final sparse architecture is both stable and high-performing.
While highly effective for convolutional filters, future research could extend this self-adaptive logic to unstructured pruning or non-convolutional architectures like Vision Transformers (ViTs), where attention heads and MLP dimensions present new challenges for adaptive compression. Despite the slight memory overhead required to keep weight backups during the search phase, PSAP offers a streamlined, effective path toward deploying high-performance DNNs on the edge.
Summary updated