To deploy deep neural networks on resource-constrained edge devices, we must compress them. However, the traditional workflow—consisting of full pre-training, structural pruning, and extensive fine-tuning—presents a massive computational bottleneck. This multi-stage overhead arises because compression is historically treated as an afterthought, decoupled from the initial training process.
When we aggressively prune or quantize a fully trained model, we disrupt its delicately optimized internal representations, forcing a costly "retraining tax" to recover lost performance. As Zhanyu Wang demonstrates in Directional Pruning of Deep Neural Networks, standard magnitude-based pruning often degrades model accuracy because it ignores the local geometry of the loss landscape, necessitating time-consuming recovery cycles. To eliminate this bottleneck, a prominent research direction unifies training and compression into a single, cohesive process.
Pruning at Initialization: Bypassing Training Entirely
If the ultimate goal is a compact, efficient network, why spend immense compute training an over-parameterized model just to throw most of it away? One elegant solution is to prune the network before training even begins.
In Snip: single-Shot Network Pruning based on Connection sensitivity, Namhoon Lee et al. introduced a method to identify and remove redundant connections at initialization. By evaluating a "connection sensitivity" criterion based on initial gradients, the authors can strip away up to 99% of the weights in a single shot. Because the model is sparse from day one, this approach completely eliminates the multi-stage train-prune-retrain pipeline, saving immense computational resources.
In-Flight Sparsification: Compressing While Learning
Another strategy integrates compression directly into the training loop, allowing the network to adapt its internal representations dynamically as it learns. Instead of a hard post-processing cut, Sharan Narang et al. showed in Exploring Sparsity in Recurrent Neural Networks that gradually zeroing out weights during the main training phase allows the model to smoothly redistribute information to the remaining parameters.
To prevent the permanent loss of network capacity early in training, Guoliang Kang et al. proposed Soft Filter Pruning (Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks). Rather than freezing pruned weights to zero, their approach allows pruned filters to be updated during the optimization process, maintaining the model's capacity to learn while continuously steering it toward a compact architecture. Similarly, Wenqi Shao et al. developed BWCP (BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening), which uses a batch whitening mechanism to amplify important channels and a differentiable Gumbel-Softmax sampler to prune channels from scratch, eliminating post-hoc fine-tuning entirely.
Continuous Optimization: Differentiable Compression
Traditional pruning is fundamentally discrete: a weight or channel is either kept ($1$) or dropped ($0$). This discrete nature prevents the use of standard gradient descent for architecture selection, forcing researchers into slow, iterative trial-and-error validation cycles.
By transforming discrete pruning into a continuous, differentiable optimization problem, we can let gradient descent do the heavy lifting. In DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation, Xuefei Ning et al. treated the sparsity budget across layers as a learnable parameter. This allows the network to automatically allocate sparsity where it hurts performance the least.
To streamline hardware deployment further, Ying Wang et al. introduced DJPQ (Differentiable Joint Pruning and Quantization for Hardware Efficiency), which optimizes both pruning and quantization simultaneously in a single training pass. By unifying these compression dimensions into a single gradient-based framework, the authors eliminate the compounding latency of sequential optimization stages.
Instant Mathematical Recovery
When post-training compression is unavoidable, can we bypass the retraining phase entirely? Recent insights show that instead of using gradient descent to slowly repair a fractured network, we can use analytical projections to instantly restore performance.
In Subspace Node Pruning, Joshua Offergeld et al. project neural network activations into an orthogonal subspace to eliminate redundancy. Crucially, instead of running dozens of fine-tuning epochs to adjust the remaining nodes, they use linear least squares to mathematically recover the lost information from the pruned nodes instantly. This analytical recovery allows their method to execute up to 24x faster than traditional iterative optimization techniques while maintaining state-of-the-art accuracy.
Key Foundational Work
Exploring Sparsity in Recurrent Neural Networks (2017) — contributed 1 concept elements to this direction.
On Compressing Deep Models by Low Rank and Sparse Decomposition (2017) — contributed 1 concept elements to this direction.
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks (2018) — contributed 1 concept elements to this direction.