TL;DR
SRigL (Structured RigL) is a new dynamic sparse training method that enforces a "constant fan-in" constraint, ensuring every neuron has the same number of active connections. By combining this hardware-friendly structure with a novel neuron ablation strategy, SRigL matches the accuracy of state-of-the-art unstructured sparse models while delivering massive real-world speedups—up to 3.4x on CPUs and 13x on GPUs—without requiring specialized hardware.
What Problem Does This Paper Address?
Neural network sparsity is a classic "no free lunch" scenario. You can have unstructured sparsity, where weights are removed anywhere in the model; this preserves accuracy but is a nightmare for hardware to accelerate because of irregular memory access. Alternatively, you can use structured sparsity (like removing entire rows or blocks), which hardware loves, but this often strips away important weights and tanks the model's performance.
Dynamic Sparse Training (DST) methods like RigL have recently shown that we can learn highly accurate sparse models from scratch. However, RigL produces unstructured masks. The authors of this paper ask: Can we get the "best of both worlds"—the accuracy of dynamic unstructured training with the hardware efficiency of structured sparsity?
Methods
The authors propose SRigL, which builds on the RigL algorithm (which periodically prunes small weights and regrows them where gradients are largest). SRigL introduces two key modifications:
- Constant Fan-In Constraint: Instead of pruning the smallest weights across an entire layer, SRigL ensures that every single neuron (or convolutional filter) maintains exactly the same number of incoming weights. This creates a highly regular structure that can be stored in a "condensed" format, making matrix multiplication much faster.
- Dynamic Neuron Ablation: The authors discovered that at very high sparsity (>90%), unstructured methods like RigL naturally "kill off" (ablate) entire neurons that aren't useful. A strict constant fan-in constraint prevents this, forcing every neuron to stay alive even if it only has 1 or 2 weak weights. SRigL introduces a threshold ($\gamma_{sal}$) that allows the model to ablate underperforming neurons and redistribute their "weight budget" to more important ones.
Datasets
The researchers evaluated SRigL across a wide range of standard benchmarks and architectures:
- CIFAR-10: Using ResNet-18 and Wide ResNet-22.
- ImageNet: Using ResNet-50, MobileNet-V3, and Vision Transformers (ViT-B/16).
Key Insights and Ideas
- Stability Matters: The authors provide a theoretical proof (Appendix A) showing that constant fan-in layers have lower output-norm variance than unstructured ones. This suggests that structured sparsity isn't just a hardware trick—it might actually make the training process more stable.
- The "Aha!" of Neuron Ablation: A major contribution is identifying that RigL's success at high sparsity comes from its ability to dynamically shrink the width of the network. By explicitly allowing SRigL to ablate neurons (as shown in Figure 2), they closed the accuracy gap that previously plagued structured DST methods.
- Hardware-Agnostic Acceleration: While NVIDIA's Ampere architecture supports 2:4 structured sparsity, SRigL's "condensed" representation allows for acceleration on any standard CPU or GPU using a simple parallel algorithm.
Key Results
SRigL consistently matches or exceeds the accuracy of RigL while providing significant hardware benefits:
- Accuracy: On ImageNet with ResNet-50, SRigL achieves 75.0% Top-1 accuracy at 80% sparsity, virtually identical to RigL’s 74.9%.
- CPU Speedup: For single-input "online" inference on an Intel Xeon, a 90% sparse SRigL layer is 3.4x faster than a dense layer and 2.5x faster than a standard unstructured sparse (CSR) layer.
- GPU Speedup: For batched inference on an NVIDIA Titan V, the same 90% sparse layer is 1.7x faster than dense and a staggering 13.0x faster than the unstructured baseline.
- Transformers: SRigL successfully sparsifies ViT-B/16, which is notoriously difficult to prune without significant accuracy loss.
Limitations and Future Work
- Hyperparameter Sensitivity: While the neuron ablation threshold ($\gamma_{sal}$) worked well at 30% for CNNs, Vision Transformers required a much higher threshold (95%), suggesting the optimal setting is architecture-dependent.
- Software Optimization: The current speedups were achieved with a relatively straightforward PyTorch/CUDA implementation. The authors suggest that custom-engineered kernels could push these performance gains even further.
- Training Costs: While SRigL reduces FLOPs during training, it still requires calculating gradients for pruned weights to decide where to regrow them, a common overhead in DST methods.
Summary updated