Introduction
Deep Neural Networks (DNNs) have revolutionized fields from computer vision to natural language processing. However, their deployment is often hindered by their massive memory and computational requirements. State-of-the-art models are frequently over-parameterized, containing millions of connections that contribute little to the final output but significantly increase the hardware burden.
Traditional pruning methods—which remove these unnecessary weights—typically rely on a "train-prune-retrain" cycle. While effective, this iterative process is computationally expensive and can double the total training time. This paper presents a novel algorithm that sparsifies networks during the initial training phase by evaluating both weight magnitudes and their gradients, achieving significant compression with minimal accuracy loss.
Methodology
The proposed Gradient and Magnitude Based Pruning algorithm introduces a two-factor criterion to identify redundant weights. Unlike magnitude-only pruning, which may inadvertently remove small but active weights, this approach considers the "stability" of a weight through its gradient.
Pruning Criteria
A weight $w$ is pruned if it satisfies two conditions simultaneously:
- Magnitude ($|w| < \beta$): The weight value is below a predefined threshold $\beta$, indicating it has a low impact on the layer's output.
- Gradient ($|g| < \gamma$): The gradient of the weight (calculated against a validation set) is below a threshold $\gamma$. A low gradient suggests the weight has reached a resting state and is no longer being actively updated by the optimizer.
Implementation via Bitmasking
To maintain sparsity during training without the need for specialized sparse optimizers, the algorithm utilizes a bitmask tensor $B$. When a weight meets the pruning criteria, its corresponding bit in $B$ is set to zero. During backpropagation, the gradients are multiplied by this bitmask (a Hadamard product), ensuring that pruned weights remain at zero and are not updated in subsequent epochs.
Experimental Setup
The researchers evaluated the method using the MobileNet architecture on the CIFAR-10 dataset. MobileNet was specifically selected as a baseline because it is already optimized for resource-constrained environments, making further compression a significant challenge. The pruning was performed iteratively according to a schedule $\delta$, with the pruning step itself implemented using NumPy’s vectorized operations for high efficiency.
Key Results and Discussion
The experimental results demonstrate that the dual-threshold approach effectively identifies "dead weight" without compromising the network's learning capacity.
Compression and Sparsity
The algorithm achieved a massive reduction in parameters, shrinking the MobileNet model from 14 million to 1.5 million parameters—a 9x compression ratio. The best-performing configuration reached a final sparsity of 89.1%.
Accuracy Trade-offs
As shown in the performance analysis, the choice of thresholds $\beta$ and $\gamma$ is critical. While aggressive pruning initially causes a dip in performance, the network is able to recover and stabilize. The optimized model achieved a Top-1 accuracy of 78.2%, representing only a 3.8% drop from the unpruned baseline (82%). Notably, this performance exceeds previous hybrid pruning methods for MobileNet, which often struggle to maintain accuracy above 74% at similar sparsity levels.
Conclusion and Future Work
By integrating pruning into the training flow, this method eliminates the overhead of post-training recovery cycles. The inclusion of gradient information provides a more robust proxy for weight importance than magnitude alone.
Future research will focus on automating the selection of the $\beta$ and $\gamma$ thresholds, which currently require manual tuning. Additionally, scaling the experiments to larger datasets like ImageNet and exploring hardware-level acceleration for the resulting sparse matrices will be essential for real-world deployment on edge devices.
Summary updated