Feather: An Elegant Solution to Effective DNN Sparsification

Athanasios Glentis Georgoulakis, George Retsinas, Petros Maragos

Abstract

Feather is a lightweight sparse training module designed to achieve state-of-the-art accuracy at extreme sparsity levels (up to 99%). By introducing a novel thresholding operator that balances the continuity of soft thresholding with the low bias of hard thresholding, and incorporating gradient scaling to stabilize mask updates, Feather enables robust neural network compression without the need for complex, multi-stage retraining cycles.

Introduction

Deep Neural Networks (DNNs) are notoriously over-parametrized, presenting significant challenges for deployment on resource-constrained edge devices like smartphones or IoT sensors. While pruning—the removal of unnecessary weights—is the standard solution, it often incurs a "performance tax." Traditional "train-prune-fine-tune" cycles are computationally expensive, leading the field toward sparse training using a Straight-Through Estimator (STE). However, existing STE methods face a critical dilemma: Hard Thresholding (abruptly cutting weights) causes training instability, while Soft Thresholding (gradually shrinking weights) introduces a constant bias that degrades final accuracy.

The Feather Module

The Feather Module addresses this gap through a mathematically elegant operator that balances stability and bias. It modifies the standard STE pipeline in two key ways:

  1. The Feather Thresholding Operator: Instead of a binary choice between hard or soft thresholding, Feather introduces a family of operators controlled by a power parameter $p$. The authors identified $p=3$ as the "Goldilocks" value: it is continuous (like soft thresholding) to prevent training jumps, but reduces bias (like hard thresholding) for weights far from the threshold.
  2. Gradient Scaling: To prevent the pruning mask from jittering at extreme sparsity levels (>95%), Feather introduces a scaling factor $\theta$. By scaling the gradients of pruned weights (typically by 0.5), the updates are dampened, allowing the network to settle into a stable, high-performing sparse structure.
The Feather Module architecture (a) and the mathematical continuity-bias tradeoff of the thresholding operator (b).

Experimental Results

Feather was evaluated across standard benchmarks, including CIFAR-100 and ImageNet, consistently outperforming existing methods at high sparsity ratios.

Large-Scale Performance

On ImageNet using ResNet-50, Feather achieved a remarkable 68.85% Top-1 accuracy at 99% sparsity. This significantly outperforms previous state-of-the-art methods like ST-3 (63.88%) and STR (51.82%).

Architectural Robustness

Ablation studies on CIFAR-100 using ResNet-20, MobileNetV1, and DenseNet40-24 confirmed Feather's superiority. At 90% sparsity, Feather-trained models occasionally achieved higher accuracy than their dense counterparts, suggesting that the sparsification process can serve as a beneficial form of regularization.

Accuracy vs. Sparsity Ratio for ResNet-20 on CIFAR-100, showing Feather's (Proposed) superior performance at high sparsity levels.

Discussion and Conclusion

The success of Feather lies in its simplicity and its focus on the mathematical "shape" of the thresholding function. By ensuring continuity while minimizing bias, Feather allows weights to transition smoothly between pruned and active states. Furthermore, gradient scaling acts as a crucial stabilizer for the extreme sparsity required by edge devices.

Despite its focus on unstructured pruning, Feather's efficiency—adding virtually zero computational overhead to the training process—makes it a highly scalable solution for modern DNN compression. Future work may explore structured variants or per-layer hyperparameter scheduling to further optimize performance for specific hardware backbones.

Summary updated

Abstract

Neural Network pruning is an increasingly popular way for producing compact and efficient models, suitable for resource-limited environments, while preserving high performance. While the pruning can be performed using a multi-cycle training and fine-tuning process, the recent trend is to encompass the sparsification process during the standard course of training. To this end, we introduce Feather, an efficient sparse training module utilizing the powerful Straight-Through Estimator as its core, coupled with a new thresholding operator and a gradient scaling technique, enabling robust, out-of-the-box sparsification performance. Feather's effectiveness and adaptability is demonstrated using various architectures on the CIFAR dataset, while on ImageNet it achieves state-of-the-art Top-1 validation accuracy using the ResNet-50 architecture, surpassing existing methods, including more complex and computationally heavy ones, by a considerable margin. Code is publicly available at https://github.com/athglentis/feather .
BibTeX
@inproceedings{georgoulakis2023feather,
  author = {Athanasios Glentis Georgoulakis and George Retsinas and Petros Maragos},
  title = {{Feather: An Elegant Solution to Effective DNN Sparsification}},
  booktitle = {BMVC},
  year = {2023},
  url = {https://openreview.net/forum?id=raERdz9BiZ},
}
Figure