Compression-aware Training of Neural Networks using Frank-Wolfe

Max Zimmer, Christoph Spiegel, Sebastian Pokutta

Introduction

Modern deep neural networks (DNNs) are often significantly over-parameterized, leading to high computational costs and memory requirements that hinder deployment on edge devices. Common compression techniques include pruning (removing weights or filters) and low-rank decomposition (approximating weight matrices with lower-rank counterparts). Traditionally, these involve a "prune-then-retrain" cycle or training a specific model for a fixed sparsity level. This paper proposes a "compression-aware" training framework using the Stochastic Frank-Wolfe (SFW) algorithm. The goal is to produce a single dense model that remains accurate even when "snapped" to various compression levels without further retraining.

Methodology: Stochastic Frank-Wolfe for Structured Compression

The core of the proposed framework, SparseFW, is the application of the Stochastic Frank-Wolfe algorithm. Unlike Stochastic Gradient Descent (SGD), which requires projection steps to enforce constraints, SFW is projection-free. It finds update directions by solving a linear minimization over a constraint set, naturally promoting structural properties.

The authors extend this to structured compression using specialized norm constraints:

  • Group-k-support norms for filter pruning: These encourage the model to concentrate information within a subset of $k$ filters.
  • Spectral-k-support norms for low-rank decomposition: These target the singular values of weight matrices, ensuring information is preserved in a low-rank subspace.

A significant technical contribution is the introduction of Gradient Rescaling, a dynamic learning rate adjustment that maintains the "pressure" for compression stability throughout the training process, even as the loss decreases.

Experimental Setup

The framework was evaluated across diverse computer vision tasks:

  • Image Classification: CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet-1K.
  • Semantic Segmentation: CityScapes using the PSPNet architecture.

Key Results and Performance

SparseFW demonstrates superior performance across both pruning and low-rank decomposition tasks.

Filter Pruning on ImageNet

On the large-scale ImageNet-1K dataset using a ResNet-50 architecture, SparseFW maintains high accuracy even as the fraction of pruned filters increases, outperforming specialized methods like SSL and ABFP.

Test accuracy of ResNet-50 on ImageNet after filter pruning. SparseFW maintains higher accuracy at higher pruning ratios compared to other methods.

Low-Rank Decomposition and Versatility

The method's effectiveness for low-rank decomposition is shown across multiple datasets. By using spectral-k-support norms and efficient partial SVD updates, SparseFW achieves significant computational gains—up to 2x to 3x higher throughput compared to nuclear-norm regularization (NUC).

Performance of SparseFW for low-rank decomposition across WideResNet (CIFAR-100), PSPNet (CityScapes), and ResNet-50 (TinyImageNet). SparseFW consistently outperforms or matches baselines across varying fractions of singular values pruned.

Discussion and Conclusion

SparseFW offers a robust alternative to traditional compression workflows by decoupling sparsity levels from model scale during training. This allows for a single training run to produce a model adaptable to various deployment constraints.

While highly effective, the current approach uses uniform sparsity across all layers. Future research could explore non-uniform, layer-wise pruning schedules. Additionally, while SFW is computationally efficient due to partial SVDs, its implementation is more involved than standard SGD. Nonetheless, SparseFW represents a significant step toward efficient, "one-size-fits-many" neural network training for resource-constrained environments.

Summary updated

BibTeX
@inproceedings{zimmer2023compression,
  author = {Max Zimmer and Christoph Spiegel and Sebastian Pokutta},
  title = {{Compression-aware Training of Neural Networks using Frank-Wolfe}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ueEMZjY9WiM},
}
Figure