Abstract
Pruning deep neural networks is a critical technique for reducing model size and computational overhead. However, traditional pruning methods face two significant hurdles: the high computational cost of identifying optimal sparse structures and the lack of hardware acceleration for unstructured sparsity. We introduce Early Compression via Gradient Flow Preservation (EarlyCroP), a method that extracts high-performing sparse models early in the training process. By preserving the network's gradient flow, EarlyCroP enables structured pruning that reduces GPU memory requirements and carbon emissions without sacrificing accuracy. Our empirical results across computer vision, natural language processing, and reinforcement learning demonstrate that EarlyCroP consistently outperforms existing pruning baselines, often matching or exceeding the performance of dense models.
Introduction
Deep neural networks are notoriously over-parameterized. The "Lottery Ticket Hypothesis" (LTH) posits that dense networks contain sparse "winning tickets"—subnetworks that, when trained in isolation, can achieve performance comparable to the original dense model. Despite the promise of LTH, the practical application of pruning remains limited by two factors. First, finding these winning tickets typically requires training the full dense model to completion, which is computationally expensive. Second, most pruning research focuses on unstructured methods (removing individual weights), which do not translate to actual speedups or memory savings on standard GPU hardware.
EarlyCroP addresses these gaps by identifying redundant structures early in the training phase and applying structured pruning to physically shrink the model tensors. This approach allows researchers to train large-scale models on commodity hardware that would otherwise be insufficient for dense training.
Methodology: The EarlyCroP Framework
EarlyCroP optimizes the pruning process through three core innovations: the pruning criterion, the structural approach, and the timing of the intervention.
1. Gradient Flow Preservation
Rather than pruning based on weight magnitude—a common but often suboptimal heuristic—EarlyCroP utilizes a sensitivity criterion designed to preserve the network's Gradient Flow (GF) and Neural Tangent Kernel (NTK). By ensuring that the sparse subnetwork maintains the same learning dynamics as the dense model, EarlyCroP allows the pruned network to follow an optimized learning path from the outset.
2. Structured Pruning (EarlyCroP-S)
To bridge the "hardware gap," EarlyCroP-S removes entire neurons or channels rather than isolated weights. This physical reduction in tensor size leads to immediate improvements in GPU RAM utilization, training speed, and energy efficiency. This makes the method "hardware-aware," enabling the training of massive architectures on limited hardware resources.
3. Identifying the "Lazy Kernel" Regime
A critical challenge in early pruning is determining when to prune. Pruning at initialization often results in poor accuracy because the network has not yet stabilized. EarlyCroP identifies a "lazy kernel regime"—a point early in training where the network's learning dynamics become approximately constant. The authors developed a novel detection score based on the relative change in weight norms to automatically trigger pruning at this optimal moment.
Experimental Results
The authors evaluated EarlyCroP across diverse domains, including Computer Vision (CIFAR, ImageNet), NLP (BERT on GLUE tasks), and Reinforcement Learning (Classic Control).
- Accuracy and Sparsity: On CIFAR-100 using a VGG16 architecture, EarlyCroP-S achieved 93.0% accuracy at 98% sparsity. Notably, this outperformed the dense baseline (90.2%) while using a fraction of the memory.
- Resource Efficiency: The method demonstrated a reduction in carbon emissions by up to 70%. Furthermore, the search for sparse models was approximately 5x less expensive than the iterative process required by the original LTH method.
- Scaling to Large Models: EarlyCroP successfully trained a ResNext101 model (829 million parameters) on a commodity GPU. The resulting pruned model was 7x faster to train and 192x smaller on disk than its dense counterpart, demonstrating the method's potential for democratizing large-scale AI research.
Discussion and Limitations
While EarlyCroP represents a significant step forward, some challenges remain. The timing of the pruning is governed by a threshold ($th = 1 - \rho$) which, though theoretically grounded, may require tuning for novel or exotic architectures. Additionally, the method still requires a brief initial period of dense training. For the largest possible models, even these few initial epochs may present a memory bottleneck before the pruning is triggered.
Conclusion
EarlyCroP provides a robust framework for efficient neural network pruning by focusing on the preservation of gradient flow early in the training cycle. By prioritizing structured sparsity and optimal timing, it offers a practical path toward reducing the environmental and financial costs of deep learning, making high-performance modeling accessible to researchers with standard hardware resources.
Summary updated