Introduction
Modern deep neural networks (DNNs) are often significantly over-parameterized, leading to high computational costs and memory requirements that hinder deployment on edge devices. Common compression techniques include pruning (removing weights or filters) and low-rank decomposition (approximating weight matrices with lower-rank counterparts). Traditionally, these involve a "prune-then-retrain" cycle or training a specific model for a fixed sparsity level. This paper proposes a "compression-aware" training framework using the Stochastic Frank-Wolfe (SFW) algorithm. The goal is to produce a single dense model that remains accurate even when "snapped" to various compression levels without further retraining.
Methodology: Stochastic Frank-Wolfe for Structured Compression
The core of the proposed framework, SparseFW, is the application of the Stochastic Frank-Wolfe algorithm. Unlike Stochastic Gradient Descent (SGD), which requires projection steps to enforce constraints, SFW is projection-free. It finds update directions by solving a linear minimization over a constraint set, naturally promoting structural properties.
The authors extend this to structured compression using specialized norm constraints:
- Group-k-support norms for filter pruning: These encourage the model to concentrate information within a subset of $k$ filters.
- Spectral-k-support norms for low-rank decomposition: These target the singular values of weight matrices, ensuring information is preserved in a low-rank subspace.
A significant technical contribution is the introduction of Gradient Rescaling, a dynamic learning rate adjustment that maintains the "pressure" for compression stability throughout the training process, even as the loss decreases.
Experimental Setup
The framework was evaluated across diverse computer vision tasks:
- Image Classification: CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet-1K.
- Semantic Segmentation: CityScapes using the PSPNet architecture.
Key Results and Performance
SparseFW demonstrates superior performance across both pruning and low-rank decomposition tasks.
Filter Pruning on ImageNet
On the large-scale ImageNet-1K dataset using a ResNet-50 architecture, SparseFW maintains high accuracy even as the fraction of pruned filters increases, outperforming specialized methods like SSL and ABFP.
Low-Rank Decomposition and Versatility
The method's effectiveness for low-rank decomposition is shown across multiple datasets. By using spectral-k-support norms and efficient partial SVD updates, SparseFW achieves significant computational gains—up to 2x to 3x higher throughput compared to nuclear-norm regularization (NUC).
Discussion and Conclusion
SparseFW offers a robust alternative to traditional compression workflows by decoupling sparsity levels from model scale during training. This allows for a single training run to produce a model adaptable to various deployment constraints.
While highly effective, the current approach uses uniform sparsity across all layers. Future research could explore non-uniform, layer-wise pruning schedules. Additionally, while SFW is computationally efficient due to partial SVDs, its implementation is more involved than standard SGD. Nonetheless, SparseFW represents a significant step toward efficient, "one-size-fits-many" neural network training for resource-constrained environments.
Summary updated