Compression-aware training of neural networks using Frank–Wolfe

Max Zimmer, Christoph Spiegel, Sebastian PokuttaOpenalex Unknown 2025 · DOI · OpenAlex

Abstract

Many existing Neural Network (NN) pruning approaches rely on either retraining or inducing a strong bias to converge to a sparse solution. A third paradigm, "compression-aware" training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single training run while avoiding retraining. We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank–Wolfe (SFW) algorithm to encourage convergence to solutions robust to filter pruning and low-rank matrix decomposition. Our method outperforms existing compression-aware approaches and, for low-rank decomposition, requires significantly fewer computational resources than nuclear-norm regularization. We also establish that dynamically adjusting the learning rate of SFW via gradient rescaling is crucial for convergence and robustness.

Introduction

Modern deep learning is characterized by a "bigger is better" philosophy, yet deploying billion-parameter models on edge devices remains a significant logistical challenge. Traditionally, this is addressed through pruning: training a model, removing the least important parameters, and then retraining to recover lost accuracy. However, this cycle is computationally expensive and must be repeated for every desired sparsity level.

In "Compression-aware training of neural networks using Frank–Wolfe," Zimmer et al. introduce a framework that allows for the training of a single dense model that remains high-performing across a wide range of compression ratios—all without a single second of retraining. This "train once, compress many ways" approach offers a more flexible path to model deployment.

The Compression-Aware Paradigm

Most network compression follows one of two paths:

  1. Post-training pruning: Pruning a pre-trained model and then fine-tuning/retraining it. This is expensive and specific to one sparsity level.
  2. Pruning during training: Baking sparsity into the training process, which usually requires picking a specific target sparsity level upfront.

The authors instead focus on the compression-aware paradigm. The goal is to produce a single dense model that is inherently stable under pruning. Once trained, a developer can remove 30%, 60%, or 90% of the filters at deployment time, and the model maintains its performance without further optimization.

Methodology: Stochastic Frank–Wolfe and k-support Norms

The core of the proposed method, dubbed SparseFW, is the shift from standard Stochastic Gradient Descent (SGD) to the Stochastic Frank–Wolfe (SFW) algorithm, combined with specialized constraints.

1. Projection-Free Optimization

Unlike SGD, which takes a step and then "projects" weights back into a feasible constraint set (a process that can be computationally intensive and mathematically messy), SFW is projection-free. In each step, it solves a Linear Minimization Oracle (LMO) to find the optimal direction within a constraint shape to move toward. This naturally encourages the model to explore the "corners" of the constraint set, which correspond to sparse or low-rank solutions.

2. The k-support Norms

To induce structural robustness, the authors utilize two specific norm constraints:

  • Group-k-support norm: Targeted at structured filter pruning, this encourages the model to concentrate its "energy" into a subset of $k$ filters.
  • Spectral-k-support norm: Targeted at low-rank decomposition, this targets the singular values of weight matrices, making the model robust to being decomposed into smaller, faster matrix products.

Unlike previous "k-sparse" constraints that forced all active weights to have the same magnitude, the k-support norm allows weights to maintain their relative importance, leading to significantly better convergence.

3. Gradient Rescaling

A critical discovery in this work is the necessity of gradient rescaling. By dynamically adjusting the learning rate based on the norm of the batch gradient, the authors decouple regularization strength from the step size. This prevents optimization collapse and ensures the weights move effectively toward sparse solutions. The authors provide the first convergence proof for SFW with gradient rescaling in a non-convex stochastic setting.

Experimental Evaluation

The researchers evaluated SparseFW across several standard benchmarks, including ImageNet-1K, TinyImageNet, CIFAR-10/100, and CityScapes (using PSPNet). They tested architectures ranging from ResNet-18 and ResNet-50 to WideResNet-28x10.

Key Findings

  • Superior Pruning Stability: On CIFAR-10 with a ResNet-18 architecture, SparseFW maintained approximately 90% accuracy even after 80% of filters were pruned. In contrast, the standard baseline dropped to ~14% accuracy at the same pruning level.
  • Efficiency Gains: For low-rank decomposition, SparseFW is significantly faster than competing methods. It achieved 2x to 3x higher training throughput (images per second) compared to nuclear-norm regularization because it only requires computing a partial SVD (k-SVD) rather than a full one.
  • Hyperparameter Robustness: The method proved less sensitive to the choice of $k$ and radius $ au$ than previous SFW-based approaches, making it easier to tune for different tasks.

Limitations and Future Work

While SparseFW excels in the "no-retraining" scenario, the authors note a "retraining gap": if extensive retraining is permitted after pruning, standard SGD-based methods can eventually reach similar accuracy levels. Additionally, performance drops were more pronounced in semantic segmentation tasks using pretrained backbones, suggesting a potential conflict between pretrained weights and the structural constraints imposed by SFW.

Future work may involve using more sophisticated randomized SVD algorithms to further boost efficiency and exploring the application of SparseFW to even larger transformer-based architectures.

Conclusion

By moving away from the "train-prune-repeat" cycle, this research paves the way for more flexible, deployment-ready AI. SparseFW allows developers to train a model once and deploy it across a variety of hardware constraints, from high-power servers to resource-constrained smartphones, without sacrificing performance or computational efficiency during the training phase.

Summary updated

Abstract

Many existing Neural Network (NN) pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training.A third paradigm, "compression-aware" training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining.We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank-Wolfe (SFW) algorithm that encourage convergence to well-performing solutions while inducing robustness towards filter pruning and low-rank matrix decomposition.Our method outperforms existing compression-aware approaches and, in the case of low-rank matrix decomposition, it also requires significantly less computational resources than approaches based on nuclear-norm regularization.Our findings indicate that dynamically adjusting the learning rate of SFW, as suggested by Pokutta et al. [57], is crucial for convergence and robustness of SFW-trained models and we establish a theoretical foundation for that practice.
BibTeX
@inproceedings{zimmer2025compression,
  author = {Max Zimmer and Christoph Spiegel and Sebastian Pokutta},
  title = {{Compression-aware training of neural networks using Frank–Wolfe}},
  booktitle = {Openalex Unknown},
  year = {2025},
  doi = {10.1515/9783111376776-010},
}
Figure