Dynamic Sparse Training with Structured Sparsity

Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, Yani Ioannou

TL;DR

SRigL (Structured RigL) is a new dynamic sparse training method that enforces a "constant fan-in" constraint, ensuring every neuron has the same number of active connections. By combining this hardware-friendly structure with a novel neuron ablation strategy, SRigL matches the accuracy of state-of-the-art unstructured sparse models while delivering massive real-world speedups—up to 3.4x on CPUs and 13x on GPUs—without requiring specialized hardware.

What Problem Does This Paper Address?

Neural network sparsity is a classic "no free lunch" scenario. You can have unstructured sparsity, where weights are removed anywhere in the model; this preserves accuracy but is a nightmare for hardware to accelerate because of irregular memory access. Alternatively, you can use structured sparsity (like removing entire rows or blocks), which hardware loves, but this often strips away important weights and tanks the model's performance.

Dynamic Sparse Training (DST) methods like RigL have recently shown that we can learn highly accurate sparse models from scratch. However, RigL produces unstructured masks. The authors of this paper ask: Can we get the "best of both worlds"—the accuracy of dynamic unstructured training with the hardware efficiency of structured sparsity?

Methods

The authors propose SRigL, which builds on the RigL algorithm (which periodically prunes small weights and regrows them where gradients are largest). SRigL introduces two key modifications:

  1. Constant Fan-In Constraint: Instead of pruning the smallest weights across an entire layer, SRigL ensures that every single neuron (or convolutional filter) maintains exactly the same number of incoming weights. This creates a highly regular structure that can be stored in a "condensed" format, making matrix multiplication much faster.
  2. Dynamic Neuron Ablation: The authors discovered that at very high sparsity (>90%), unstructured methods like RigL naturally "kill off" (ablate) entire neurons that aren't useful. A strict constant fan-in constraint prevents this, forcing every neuron to stay alive even if it only has 1 or 2 weak weights. SRigL introduces a threshold ($\gamma_{sal}$) that allows the model to ablate underperforming neurons and redistribute their "weight budget" to more important ones.
Figure 1: (a) Constant fan-in pruning keeps the most salient weights per neuron, while unstructured pruning keeps the most salient weights per layer. A constant fan-in weight matrix has the same number of non-zero elements (here 2) per column allowing condensed representation. While pruning may remove salient weights affecting gen- eralization, with SRigL structure and weights are learned concurrently. (b) Output-norm variance: Theoretical predictions and simulation results (see Appendix A) demonstrating that sparse layers with constant fan-in have consistently smaller output-norm variance than layers with the same sparsity but w/o the constant fan-in constraint.

Datasets

The researchers evaluated SRigL across a wide range of standard benchmarks and architectures:

  • CIFAR-10: Using ResNet-18 and Wide ResNet-22.
  • ImageNet: Using ResNet-50, MobileNet-V3, and Vision Transformers (ViT-B/16).

Key Insights and Ideas

  • Stability Matters: The authors provide a theoretical proof (Appendix A) showing that constant fan-in layers have lower output-norm variance than unstructured ones. This suggests that structured sparsity isn't just a hardware trick—it might actually make the training process more stable.
  • The "Aha!" of Neuron Ablation: A major contribution is identifying that RigL's success at high sparsity comes from its ability to dynamically shrink the width of the network. By explicitly allowing SRigL to ablate neurons (as shown in Figure 2), they closed the accuracy gap that previously plagued structured DST methods.
  • Hardware-Agnostic Acceleration: While NVIDIA's Ampere architecture supports 2:4 structured sparsity, SRigL's "condensed" representation allows for acceleration on any standard CPU or GPU using a simple parallel algorithm.
Figure 2: Neuron ablation. At sparsity levels over 90%, RigL learns to completely mask (ablate) a large number of neurons within each layer, effectively reducing layer width. Imposing a constant fan-in constraint requires all neurons to have the same number of (non-pruned) incoming weights and therefore inhibits ablation, which results in worse generalization performance than RigL. Allowing SRigL to ablate neurons restores RigL-level performance.

Key Results

SRigL consistently matches or exceeds the accuracy of RigL while providing significant hardware benefits:

  • Accuracy: On ImageNet with ResNet-50, SRigL achieves 75.0% Top-1 accuracy at 80% sparsity, virtually identical to RigL’s 74.9%.
  • CPU Speedup: For single-input "online" inference on an Intel Xeon, a 90% sparse SRigL layer is 3.4x faster than a dense layer and 2.5x faster than a standard unstructured sparse (CSR) layer.
  • GPU Speedup: For batched inference on an NVIDIA Titan V, the same 90% sparse layer is 1.7x faster than dense and a staggering 13.0x faster than the unstructured baseline.
  • Transformers: SRigL successfully sparsifies ViT-B/16, which is notoriously difficult to prune without significant accuracy loss.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the neuron ablation threshold ($\gamma_{sal}$) worked well at 30% for CNNs, Vision Transformers required a much higher threshold (95%), suggesting the optimal setting is architecture-dependent.
  • Software Optimization: The current speedups were achieved with a relatively straightforward PyTorch/CUDA implementation. The authors suggest that custom-engineered kernels could push these performance gains even further.
  • Training Costs: While SRigL reduces FLOPs during training, it still requires calculating gradients for pruned weights to decide where to regrow them, a common overhead in DST methods.

Summary updated

Abstract

Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting models are highly sparse and theoretically less computationally expensive, achieving speedups with unstructured sparsity on real-world hardware is challenging. In this work, we propose a sparse-to-sparse DST method, Structured RigL (SRigL), to learn a variant of fine-grained structured N:M sparsity by imposing a constant fan-in constraint. Using our empirical analysis of existing DST methods at high sparsity, we additionally employ a neuron ablation method which enables SRigL to achieve state-of-the-art sparse-to-sparse structured DST performance on a variety of Neural Network (NN) architectures. Using a 90% sparse linear layer, we demonstrate a real-world acceleration of 3.4x/2.5x on CPU for online inference and 1.7x/13.0x on GPU for inference with a batch size of 256 when compared to equivalent dense/unstructured (CSR) sparse layers, respectively.
BibTeX
@inproceedings{lasby2024dynamic,
  author = {Mike Lasby and Anna Golubeva and Utku Evci and Mihai Nica and Yani Ioannou},
  title = {{Dynamic Sparse Training with Structured Sparsity}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=kOBkxFRKTA},
}
Figure