Abstract
Feather is a lightweight sparse training module designed to achieve state-of-the-art accuracy at extreme sparsity levels (up to 99%). By introducing a novel thresholding operator that balances the continuity of soft thresholding with the low bias of hard thresholding, and incorporating gradient scaling to stabilize mask updates, Feather enables robust neural network compression without the need for complex, multi-stage retraining cycles.
Introduction
Deep Neural Networks (DNNs) are notoriously over-parametrized, presenting significant challenges for deployment on resource-constrained edge devices like smartphones or IoT sensors. While pruning—the removal of unnecessary weights—is the standard solution, it often incurs a "performance tax." Traditional "train-prune-fine-tune" cycles are computationally expensive, leading the field toward sparse training using a Straight-Through Estimator (STE). However, existing STE methods face a critical dilemma: Hard Thresholding (abruptly cutting weights) causes training instability, while Soft Thresholding (gradually shrinking weights) introduces a constant bias that degrades final accuracy.
The Feather Module
The Feather Module addresses this gap through a mathematically elegant operator that balances stability and bias. It modifies the standard STE pipeline in two key ways:
- The Feather Thresholding Operator: Instead of a binary choice between hard or soft thresholding, Feather introduces a family of operators controlled by a power parameter $p$. The authors identified $p=3$ as the "Goldilocks" value: it is continuous (like soft thresholding) to prevent training jumps, but reduces bias (like hard thresholding) for weights far from the threshold.
- Gradient Scaling: To prevent the pruning mask from jittering at extreme sparsity levels (>95%), Feather introduces a scaling factor $\theta$. By scaling the gradients of pruned weights (typically by 0.5), the updates are dampened, allowing the network to settle into a stable, high-performing sparse structure.
Experimental Results
Feather was evaluated across standard benchmarks, including CIFAR-100 and ImageNet, consistently outperforming existing methods at high sparsity ratios.
Large-Scale Performance
On ImageNet using ResNet-50, Feather achieved a remarkable 68.85% Top-1 accuracy at 99% sparsity. This significantly outperforms previous state-of-the-art methods like ST-3 (63.88%) and STR (51.82%).
Architectural Robustness
Ablation studies on CIFAR-100 using ResNet-20, MobileNetV1, and DenseNet40-24 confirmed Feather's superiority. At 90% sparsity, Feather-trained models occasionally achieved higher accuracy than their dense counterparts, suggesting that the sparsification process can serve as a beneficial form of regularization.
Discussion and Conclusion
The success of Feather lies in its simplicity and its focus on the mathematical "shape" of the thresholding function. By ensuring continuity while minimizing bias, Feather allows weights to transition smoothly between pruned and active states. Furthermore, gradient scaling acts as a crucial stabilizer for the extreme sparsity required by edge devices.
Despite its focus on unstructured pruning, Feather's efficiency—adding virtually zero computational overhead to the training process—makes it a highly scalable solution for modern DNN compression. Future work may explore structured variants or per-layer hyperparameter scheduling to further optimize performance for specific hardware backbones.
Summary updated