Abstract
Convolutional neural networks (CNNs) have succeeded in many practical applications. However, their high computation and storage requirements often make them difficult to deploy on resource-constrained devices. In order to tackle this issue, many pruning algorithms have been proposed for CNNs, but most of them can't prune CNNs to a reasonable level. In this paper, we propose a novel algorithm for training and pruning CNNs based on the recursive least squares (RLS) optimization. After training a CNN for some epochs, our algorithm combines inverse input autocorrelation matrices and weight matrices to evaluate and prune unimportant input channels or nodes layer by layer. Then, our algorithm will continue to train the pruned network, and won't do the next pruning until the pruned network recovers the full performance of the old network. Besides for CNNs, the proposed algorithm can be used for feedforward neural networks (FNNs). Three experiments on MNIST, CIFAR-10 and SVHN datasets show that our algorithm can achieve the more reasonable pruning and have higher learning efficiency than other four popular pruning algorithms.
Introduction
Deep learning models, particularly Convolutional Neural Networks (CNNs), have revolutionized computer vision but remain notoriously resource-intensive. Their massive computational and storage requirements pose significant challenges for deployment on hardware with limited power and memory, such as smartphones, drones, or wearable devices.
Network pruning—the process of removing redundant parameters—is a common solution. However, traditional pruning methods often suffer from being heuristic-based (e.g., only considering weight magnitudes), relying on one-shot strategies that can permanently damage accuracy, and using slow first-order optimizers like SGD. This paper addresses these gaps by introducing a training and pruning framework based on Recursive Least Squares (RLS).
Methodology
The proposed approach leverages the RLS algorithm, a recursive version of linear least squares known for its fast convergence in signal processing. Unlike standard optimizers, RLS naturally maintains an inverse input autocorrelation matrix ($P_t$) during training. This matrix provides a second-order view of the error surface and contains critical information about which input components actually influence the network's output.
Dual-Metric Pruning and Adaptive Strategy
To determine which nodes or channels to remove, the algorithm employs a dual-metric approach. It combines Input Importance, derived from the $P_t$ matrix, with Weight Importance, measured by the $L_1$-norm of the weights. This ensures that a channel is only kept if both its weights and its input features are significant.
Rather than a single pruning event, the framework uses an Adaptive Multi-shot Strategy. The network is pruned in small increments. After each round, training continues until the network's Mean Squared Error (MSE) recovers to its pre-pruning level. This "healing" loop allows the model to adapt to its reduced capacity before further surgery is performed. Notably, this method can even prune redundant input features, such as unimportant pixels in an image.
Experimental Evaluation
The researchers validated the RLS-based method on three standard benchmarks: MNIST, CIFAR-10, and SVHN.
- Extreme Compression: On the MNIST dataset, the algorithm successfully retained only 16.4% of weights and 41.1% of nodes while maintaining 98.5% accuracy. It also identified that 67% of the original input pixels were redundant.
- CNN Efficiency: For the SVHN dataset, the model was compressed to just 6.5% of its original weights with a negligible accuracy drop (from 94.7% to 94.1%).
- Stability: Compared to Momentum-based methods, the RLS approach demonstrated a significantly smoother recovery of classification precision after pruning events, highlighting the efficiency of second-order optimization in the prune-and-recover cycle.
Conclusion
By integrating input statistics into the pruning decision and using an iterative recovery process, the RLS-based framework achieves superior compression ratios compared to traditional $L_1$-norm pruning or Net Slimming. While the computational complexity of RLS is higher than SGD, the use of the Sherman-Morrison formula makes it manageable for the tested architectures. Future research will focus on scaling this approach to high-resolution datasets like ImageNet and automating the selection of pruning hyperparameters.
Summary updated