Pruning Deep Neural Networks from a Sparsity Perspective

Enmao Diao, Ganghua Wang, Jiawei Zhang, Yuhong Yang, Jie Ding, Vahid Tarokh

Abstract

In recent years, deep network pruning has attracted significant attention in order to enable the rapid deployment of AI into small devices with computation and memory constraints. Pruning is often achieved by dropping redundant weights, neurons, or layers of a deep network while attempting to retain a comparable test performance. Many deep pruning algorithms have been proposed with impressive empirical success. However, existing approaches lack a quantifiable measure to estimate the compressibility of a sub-network during each pruning iteration and thus may under-prune or over-prune the model. In this work, we propose PQ Index (PQI) to measure the potential compressibility of deep neural networks and use this to develop a Sparsity-informed Adaptive Pruning (SAP) algorithm. Our extensive experiments corroborate the hypothesis that for a generic pruning procedure, PQI decreases first when a large model is being effectively regularized and then increases when its compressibility reaches a limit that appears to correspond to the beginning of underfitting. Subsequently, PQI decreases again when the model collapse and significant deterioration in the performance of the model start to occur. Additionally, our experiments demonstrate that the proposed adaptive pruning algorithm with proper choice of hyper-parameters is superior to the iterative pruning algorithms such as the lottery ticket-based pruning methods, in terms of both compression efficiency and robustness.

Introduction

Deep neural networks are notoriously over-parameterized, often containing millions or billions of weights that contribute minimally to the final output. While network pruning—removing redundant weights—is essential for deploying AI on resource-constrained devices, it remains an empirical "black art." Most existing methods rely on fixed pruning ratios (e.g., removing 20% of weights per iteration), which fail to account for the model's actual state, leading to potential under-pruning or catastrophic model collapse. This work introduces a quantifiable, theoretically grounded measure to determine exactly how compressible a sub-network is at any given moment.

The PQ Index: A New Measure for Sparsity

We introduce the PQ Index (PQI), a norm-based measure ($L_p$ and $L_q$ norms) that quantifies the "true" sparsity of a network by examining the distribution of weight magnitudes. Unlike simple zero-counting, PQI measures how concentrated weight "energy" is. It satisfies six rigorous mathematical properties, such as the "Robin Hood" and "Bill Gates" properties, ensuring it accurately reflects functional sparsity even before weights are explicitly set to zero. Essentially, a model where most weights are tiny and a few are large is "sparser" in a functional sense than one where all weights are medium-sized.

The Three Phases of Pruning

Our research identifies a distinct lifecycle in the pruning process, characterized by the behavior of the PQI and model performance:

  1. Regularization: Pruning removes noise, often leading to a slight increase in performance while PQI decreases.
  2. Compression: The model enters a "Goldilocks zone" of maximum efficiency where sparsity increases and PQI begins to rise as the limit of compressibility is reached.
  3. Collapse: Further pruning removes essential information, causing a sharp drop in accuracy and a subsequent decrease in PQI.
Conceptual overview of the three phases of pruning: Regularization, Compression, and Collapse.

Sparsity-informed Adaptive Pruning (SAP)

Leveraging the PQI, we developed the Sparsity-informed Adaptive Pruning (SAP) algorithm. Instead of a fixed ratio, SAP uses a theoretical bound derived from the PQI to calculate an optimal "retaining ratio" for each iteration. If the PQI indicates high redundancy, SAP prunes aggressively; as the model becomes leaner, the pruning rate slows down automatically. This allows SAP to reach high compression levels in fewer iterations and with greater robustness than fixed-ratio methods like the Lottery Ticket Hypothesis.

Experimental Results

We validated SAP across standard benchmarks including FashionMNIST, CIFAR10/100, and TinyImageNet using architectures like ResNet and WideResNet.

Comparative results on FashionMNIST showing accuracy, weight retention, and PQ Index dynamics.

Key findings include:

  • Efficiency: SAP consistently finds smaller, more accurate sub-networks than "One Shot" or standard "Lottery Ticket" methods.
  • Robustness: On FashionMNIST, SAP achieved performance levels at iteration 10 that the Lottery Ticket method required 25 iterations to reach.
  • Validation: Experimental results closely followed the PQI Hypothesis, confirming PQI as a reliable "heart rate monitor" for model health during compression.

Conclusion

The PQ Index provides a theoretically grounded metric for network compressibility, enabling the adaptive SAP algorithm to outperform traditional fixed-ratio pruning. While current work provides a lower bound on parameter retention, future research will focus on identifying optimal stopping points and exploring PQI as a regularization term during initial training to encourage "prunable" models from the start.

Summary updated

Abstract

In recent years, deep network pruning has attracted significant attention in order to enable the rapid deployment of AI into small devices with computation and memory constraints. Pruning is often achieved by dropping redundant weights, neurons, or layers of a deep network while attempting to retain a comparable test performance. Many deep pruning algorithms have been proposed with impressive empirical success. However, existing approaches lack a quantifiable measure to estimate the compressibility of a sub-network during each pruning iteration and thus may under-prune or over-prune the model. In this work, we propose PQ Index (PQI) to measure the potential compressibility of deep neural networks and use this to develop a Sparsity-informed Adaptive Pruning (SAP) algorithm. Our extensive experiments corroborate the hypothesis that for a generic pruning procedure, PQI decreases first when a large model is being effectively regularized and then increases when its compressibility reaches a limit that appears to correspond to the beginning of underfitting. Subsequently, PQI decreases again when the model collapse and significant deterioration in the performance of the model start to occur. Additionally, our experiments demonstrate that the proposed adaptive pruning algorithm with proper choice of hyper-parameters is superior to the iterative pruning algorithms such as the lottery ticket-based pruning methods, in terms of both compression efficiency and robustness.
BibTeX
@inproceedings{diao2023pruning,
  author = {Enmao Diao and Ganghua Wang and Jiawei Zhang and Yuhong Yang and Jie Ding and Vahid Tarokh},
  title = {{Pruning Deep Neural Networks from a Sparsity Perspective}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=i-DleYh34BM},
}
Figure