Abstract
In recent years, deep network pruning has attracted significant attention in order to enable the rapid deployment of AI into small devices with computation and memory constraints. Pruning is often achieved by dropping redundant weights, neurons, or layers of a deep network while attempting to retain a comparable test performance. Many deep pruning algorithms have been proposed with impressive empirical success. However, existing approaches lack a quantifiable measure to estimate the compressibility of a sub-network during each pruning iteration and thus may under-prune or over-prune the model. In this work, we propose PQ Index (PQI) to measure the potential compressibility of deep neural networks and use this to develop a Sparsity-informed Adaptive Pruning (SAP) algorithm. Our extensive experiments corroborate the hypothesis that for a generic pruning procedure, PQI decreases first when a large model is being effectively regularized and then increases when its compressibility reaches a limit that appears to correspond to the beginning of underfitting. Subsequently, PQI decreases again when the model collapse and significant deterioration in the performance of the model start to occur. Additionally, our experiments demonstrate that the proposed adaptive pruning algorithm with proper choice of hyper-parameters is superior to the iterative pruning algorithms such as the lottery ticket-based pruning methods, in terms of both compression efficiency and robustness.
Introduction
Deep neural networks are notoriously over-parameterized, often containing millions or billions of weights that contribute minimally to the final output. While network pruning—removing redundant weights—is essential for deploying AI on resource-constrained devices, it remains an empirical "black art." Most existing methods rely on fixed pruning ratios (e.g., removing 20% of weights per iteration), which fail to account for the model's actual state, leading to potential under-pruning or catastrophic model collapse. This work introduces a quantifiable, theoretically grounded measure to determine exactly how compressible a sub-network is at any given moment.
The PQ Index: A New Measure for Sparsity
We introduce the PQ Index (PQI), a norm-based measure ($L_p$ and $L_q$ norms) that quantifies the "true" sparsity of a network by examining the distribution of weight magnitudes. Unlike simple zero-counting, PQI measures how concentrated weight "energy" is. It satisfies six rigorous mathematical properties, such as the "Robin Hood" and "Bill Gates" properties, ensuring it accurately reflects functional sparsity even before weights are explicitly set to zero. Essentially, a model where most weights are tiny and a few are large is "sparser" in a functional sense than one where all weights are medium-sized.
The Three Phases of Pruning
Our research identifies a distinct lifecycle in the pruning process, characterized by the behavior of the PQI and model performance:
- Regularization: Pruning removes noise, often leading to a slight increase in performance while PQI decreases.
- Compression: The model enters a "Goldilocks zone" of maximum efficiency where sparsity increases and PQI begins to rise as the limit of compressibility is reached.
- Collapse: Further pruning removes essential information, causing a sharp drop in accuracy and a subsequent decrease in PQI.
Sparsity-informed Adaptive Pruning (SAP)
Leveraging the PQI, we developed the Sparsity-informed Adaptive Pruning (SAP) algorithm. Instead of a fixed ratio, SAP uses a theoretical bound derived from the PQI to calculate an optimal "retaining ratio" for each iteration. If the PQI indicates high redundancy, SAP prunes aggressively; as the model becomes leaner, the pruning rate slows down automatically. This allows SAP to reach high compression levels in fewer iterations and with greater robustness than fixed-ratio methods like the Lottery Ticket Hypothesis.
Experimental Results
We validated SAP across standard benchmarks including FashionMNIST, CIFAR10/100, and TinyImageNet using architectures like ResNet and WideResNet.
Key findings include:
- Efficiency: SAP consistently finds smaller, more accurate sub-networks than "One Shot" or standard "Lottery Ticket" methods.
- Robustness: On FashionMNIST, SAP achieved performance levels at iteration 10 that the Lottery Ticket method required 25 iterations to reach.
- Validation: Experimental results closely followed the PQI Hypothesis, confirming PQI as a reliable "heart rate monitor" for model health during compression.
Conclusion
The PQ Index provides a theoretically grounded metric for network compressibility, enabling the adaptive SAP algorithm to outperform traditional fixed-ratio pruning. While current work provides a lower bound on parameter retention, future research will focus on identifying optimal stopping points and exploring PQI as a regularization term during initial training to encourage "prunable" models from the start.
Summary updated