Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining

Lu Miao, Xiaolong Luo, Tianlong Chen, Wuyang Chen, Dong Liu, Zhangyang WangICLR 2022 · OpenReview

In the push for "Green AI" and efficient mobile deployment, engineers rely heavily on model pruning—removing the least important weights from a large, pre-trained network to make it smaller and faster. However, standard pruning comes with a steep "retraining tax."

If you prune a model trained with standard Stochastic Gradient Descent (SGD) and do not retrain it, its accuracy usually falls off a cliff, especially at high sparsity levels. To recover this lost performance, practitioners must perform iterative cycles of pruning and fine-tuning. This process is computationally expensive, highly sensitive to hyperparameters, and must be repeated for every target device with a different memory constraint.

Researchers have recently developed SFW-pruning, a novel framework that asks a fundamental question: Can we train a network to be "born" pruning-friendly, so it can be shrunk to any size on the fly without further work? By replacing standard SGD with a "pruning-aware" optimization strategy, SFW-pruning allows a deep neural network to be trained just once and then pruned to any desired sparsity level (from 50% to 95%) without ever needing to retrain.

A New Approach: SFW-Pruning and SFW-Init

The paper introduces two major innovations to the traditional training pipeline to achieve this "one-shot, any-sparsity" capability:

  1. SFW-Pruning Framework: Instead of using SGD, the authors employ the Stochastic Frank-Wolfe (SFW) algorithm. They formulate the training process as a constrained optimization problem where the network's weights must lie within a $K$-sparse polytope. In each step of SFW, the algorithm finds a highly sparse update vector (via a "Linear Minimization Oracle"). The final weights become a weighted average of these sparse updates. This acts like a continuous "voting" process where the model gradually reaches a consensus on which weights are truly important, naturally pushing the unimportant ones toward zero.

  2. SFW-Init: Standard initialization techniques (like Kaiming or Xavier) were designed to keep gradients stable specifically for SGD. They are fundamentally misaligned with SFW, which does not utilize gradient magnitude in the same way. To solve this, the authors propose a meta-learning-based initialization scheme that learns layer-wise scaling factors to maximize the loss reduction in the very first step of SFW training.

Key Insights: Why It Works

The "Aha!" moment of this framework becomes clear when examining the resulting weight histograms of the trained models.

Weight distributions of SGD vs. SFW-trained models.
  • Smooth Weight Distributions: SGD-trained models typically exhibit a "spiky" distribution of large weights. When pruned, they hit a threshold where vital information is suddenly severed, causing accuracy to plummet. In contrast, SFW-trained models develop a much smoother distribution with a higher concentration of small-magnitude weights. This allows for a "graceful degradation" as sparsity increases.
  • Nature Sparsity: The authors discovered that SFW-trained models possess an inherent "nature sparsity." By simply adjusting the initial learning rate, practitioners can control how sparse the model naturally tends to be, even before any formal pruning is applied.

Key Results: One-Shot for Any-Sparsity

Because the model is trained with the sparse polytope constraint, the "important" weights are already well-defined by the end of the single training run. You can prune the resulting model to 10% sparsity or 90% sparsity and achieve state-of-the-art results for both, all from the same foundation.

Pruning test accuracy for ResNet18 on CIFAR-10 across different pruning ratios.

The framework was evaluated on standard computer vision benchmarks (CIFAR-10, CIFAR-100, and Tiny-ImageNet) using popular architectures like ResNet-18 and VGG-16, yielding impressive results:

  • No Retraining Required: SFW-pruning significantly outperforms one-shot magnitude pruning on SGD models. For instance, on VGG-16 (CIFAR-10), SFW-pruning maintains high accuracy even at 95% sparsity, whereas an SGD model without retraining completely collapses.
  • Competitive with Retraining: Remarkably, SFW-pruning (which uses zero retraining) achieves accuracy comparable to expensive "SGD + Retraining" methods across most pruning ratios.
  • State-of-the-Art Performance: In head-to-head comparisons with other "one-shot" or "no-retrain" methods, SFW-pruning consistently achieved higher accuracy at extreme sparsity levels (e.g., maintaining 92.00% accuracy at 95% sparsity on CIFAR-10).

Limitations and Future Directions

While SFW-pruning represents a significant leap forward for efficient model deployment, there are areas for future exploration. The current paper focuses primarily on unstructured pruning (removing individual weights). While excellent for theoretical compression, unstructured sparsity doesn't always translate to immediate latency speedups on standard hardware without specialized sparse kernels. Extending this Frank-Wolfe approach to structured pruning (removing entire neurons or channels) is a logical and highly anticipated next step.

Additionally, the framework introduces new hyperparameters, such as the radius $\tau$ and the sparsity factor $K$ of the polytope. While the authors provide robust default values, finding the absolute optimal settings for entirely new architectures might require some initial search. Finally, while SFW-Init drastically improves performance, it does add a small meta-learning overhead at the very beginning of training—though this is still vastly cheaper than the iterative retraining cycles it eliminates.

Summary updated

Abstract

We present a novel framework to train a large deep neural network (DNN) for only $\textit{once}$, which can then be pruned to $\textit{any sparsity ratio}$ to preserve competitive accuracy $\textit{without any re-training}$. Conventional methods often require (iterative) pruning followed by re-training, which not only incurs large overhead beyond the original DNN training but also can be sensitive to retraining hyperparameters. Our core idea is to re-cast the DNN training as an explicit $\textit{pruning-aware}$ process: that is formulated with an auxiliary $K$-sparse polytope constraint, to encourage network weights to lie in a convex hull spanned by $K$-sparse vectors, potentially resulting in more sparse weight matrices. We then leverage a stochastic Frank-Wolfe (SFW) algorithm to solve this new constrained optimization, which naturally leads to sparse weight updates each time. We further note an overlooked fact that existing DNN initializations were derived to enhance SGD training (e.g., avoid gradient explosion or collapse), but was unaligned with the challenges of training with SFW. We hence also present the first learning-based initialization scheme specifically for boosting SFW-based DNN training. Experiments on CIFAR-10 and Tiny-ImageNet datasets demonstrate that our new framework named $\textbf{SFW-pruning}$ consistently achieves the state-of-the-art performance on various benchmark DNNs over a wide range of pruning ratios. Moreover, SFW-pruning only needs to train once on the same model and dataset, for obtaining arbitrary ratios, while requiring neither iterative pruning nor retraining. All codes will be released to the public.
BibTeX
@inproceedings{miao2022learning,
  author = {Lu Miao and Xiaolong Luo and Tianlong Chen and Wuyang Chen and Dong Liu and Zhangyang Wang},
  title = {{Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=lnFc4ZCjTRJ},
}
Figure