Winning the Lottery Ahead of Time: Efficient Early Network Pruning

John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, Stephan Günnemann

Abstract

Pruning deep neural networks is a critical technique for reducing model size and computational overhead. However, traditional pruning methods face two significant hurdles: the high computational cost of identifying optimal sparse structures and the lack of hardware acceleration for unstructured sparsity. We introduce Early Compression via Gradient Flow Preservation (EarlyCroP), a method that extracts high-performing sparse models early in the training process. By preserving the network's gradient flow, EarlyCroP enables structured pruning that reduces GPU memory requirements and carbon emissions without sacrificing accuracy. Our empirical results across computer vision, natural language processing, and reinforcement learning demonstrate that EarlyCroP consistently outperforms existing pruning baselines, often matching or exceeding the performance of dense models.

Introduction

Deep neural networks are notoriously over-parameterized. The "Lottery Ticket Hypothesis" (LTH) posits that dense networks contain sparse "winning tickets"—subnetworks that, when trained in isolation, can achieve performance comparable to the original dense model. Despite the promise of LTH, the practical application of pruning remains limited by two factors. First, finding these winning tickets typically requires training the full dense model to completion, which is computationally expensive. Second, most pruning research focuses on unstructured methods (removing individual weights), which do not translate to actual speedups or memory savings on standard GPU hardware.

EarlyCroP addresses these gaps by identifying redundant structures early in the training phase and applying structured pruning to physically shrink the model tensors. This approach allows researchers to train large-scale models on commodity hardware that would otherwise be insufficient for dense training.

Methodology: The EarlyCroP Framework

EarlyCroP optimizes the pruning process through three core innovations: the pruning criterion, the structural approach, and the timing of the intervention.

1. Gradient Flow Preservation

Rather than pruning based on weight magnitude—a common but often suboptimal heuristic—EarlyCroP utilizes a sensitivity criterion designed to preserve the network's Gradient Flow (GF) and Neural Tangent Kernel (NTK). By ensuring that the sparse subnetwork maintains the same learning dynamics as the dense model, EarlyCroP allows the pruned network to follow an optimized learning path from the outset.

2. Structured Pruning (EarlyCroP-S)

To bridge the "hardware gap," EarlyCroP-S removes entire neurons or channels rather than isolated weights. This physical reduction in tensor size leads to immediate improvements in GPU RAM utilization, training speed, and energy efficiency. This makes the method "hardware-aware," enabling the training of massive architectures on limited hardware resources.

3. Identifying the "Lazy Kernel" Regime

A critical challenge in early pruning is determining when to prune. Pruning at initialization often results in poor accuracy because the network has not yet stabilized. EarlyCroP identifies a "lazy kernel regime"—a point early in training where the network's learning dynamics become approximately constant. The authors developed a novel detection score based on the relative change in weight norms to automatically trigger pruning at this optimal moment.

Experimental Results

The authors evaluated EarlyCroP across diverse domains, including Computer Vision (CIFAR, ImageNet), NLP (BERT on GLUE tasks), and Reinforcement Learning (Classic Control).

  • Accuracy and Sparsity: On CIFAR-100 using a VGG16 architecture, EarlyCroP-S achieved 93.0% accuracy at 98% sparsity. Notably, this outperformed the dense baseline (90.2%) while using a fraction of the memory.
  • Resource Efficiency: The method demonstrated a reduction in carbon emissions by up to 70%. Furthermore, the search for sparse models was approximately 5x less expensive than the iterative process required by the original LTH method.
  • Scaling to Large Models: EarlyCroP successfully trained a ResNext101 model (829 million parameters) on a commodity GPU. The resulting pruned model was 7x faster to train and 192x smaller on disk than its dense counterpart, demonstrating the method's potential for democratizing large-scale AI research.

Discussion and Limitations

While EarlyCroP represents a significant step forward, some challenges remain. The timing of the pruning is governed by a threshold ($th = 1 - \rho$) which, though theoretically grounded, may require tuning for novel or exotic architectures. Additionally, the method still requires a brief initial period of dense training. For the largest possible models, even these few initial epochs may present a memory bottleneck before the pruning is triggered.

Conclusion

EarlyCroP provides a robust framework for efficient neural network pruning by focusing on the preservation of gradient flow early in the training cycle. By prioritizing structured sparsity and optimal timing, it offers a practical path toward reducing the environmental and financial costs of deep learning, making high-performance modeling accessible to researchers with standard hardware resources.

Summary updated

Abstract

Pruning, the task of sparsifying deep neural networks, received increasing attention recently. Although state-of-the-art pruning methods extract highly sparse models, they neglect two main challenges: (1) the process of finding these sparse models is often very expensive; (2) unstructured pruning does not provide benefits in terms of GPU memory, training time, or carbon emissions. We propose Early Compression via Gradient Flow Preservation (EarlyCroP), which efficiently extracts state-of-the-art sparse models before or early in training addressing challenge (1), and can be applied in a structured manner addressing challenge (2). This enables us to train sparse networks on commodity GPUs whose dense versions would be too large, thereby saving costs and reducing hardware requirements. We empirically show that EarlyCroP outperforms a rich set of baselines for many tasks (incl. classification, regression) and domains (incl. computer vision, natural language processing, and reinforcment learning). EarlyCroP leads to accuracy comparable to dense training while outperforming pruning baselines.
BibTeX
@inproceedings{rachwan2022winning,
  author = {John Rachwan and Daniel Zügner and Bertrand Charpentier and Simon Geisler and Morgane Ayle and Stephan Günnemann},
  title = {{Winning the Lottery Ahead of Time: Efficient Early Network Pruning}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=wnrEwDFIHMT},
}
Figure